[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:05 dist-test-slave-1sv7 krb5kdc[9788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:05 dist-test-slave-1sv7 krb5kdc[9788](info): set up 2 sockets
Sep 25 14:55:05 dist-test-slave-1sv7 krb5kdc[9788](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:55:06.421577 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:06 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:06.551702 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45815
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:45815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:06.635607 9801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:06.635816 9801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:06.635838 9801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:06.637056 9801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:06.637096 9801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:06.637110 9801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:06.637123 9801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:06.637135 9801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:06.638626 9801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:45815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:06.638877 9801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:06.639101 9801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:06.643958 9809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:06.643996 9806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:06.644044 9801 server_base.cc:1034] running on GCE node
W20260925 14:55:06.644007 9807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:06.644446 9801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:06.644757 9801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:06.645924 9801 hybrid_clock.cc:648] HybridClock initialized: now 1790348106645903 us; error 59 us; skew 500 ppm
Sep 25 14:55:06 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:06.648488 9801 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:06.648864 9801 webserver.cc:460] Webserver started at http://127.9.141.126:46869/ using document root <none> and password file <none>
I20260925 14:55:06.649052 9801 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:06.649092 9801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:06.649199 9801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:06.650276 9801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "2c64587eaff546c0bc39ef2bded64611"
format_stamp: "Formatted at 2026-09-25 14:55:06 on dist-test-slave-1sv7"
server_key: "050180672346ff6b39b4c5f2ad71a0db"
server_key_iv: "45d1d314661db72fed68c629b2fe72d0"
server_key_version: "encryptionkey@0"
I20260925 14:55:06.650650 9801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "2c64587eaff546c0bc39ef2bded64611"
format_stamp: "Formatted at 2026-09-25 14:55:06 on dist-test-slave-1sv7"
server_key: "050180672346ff6b39b4c5f2ad71a0db"
server_key_iv: "45d1d314661db72fed68c629b2fe72d0"
server_key_version: "encryptionkey@0"
I20260925 14:55:06.651993 9801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:06.652782 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:06.652951 9801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:06.653054 9801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "2c64587eaff546c0bc39ef2bded64611"
format_stamp: "Formatted at 2026-09-25 14:55:06 on dist-test-slave-1sv7"
server_key: "050180672346ff6b39b4c5f2ad71a0db"
server_key_iv: "45d1d314661db72fed68c629b2fe72d0"
server_key_version: "encryptionkey@0"
I20260925 14:55:06.653136 9801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:06.669384 9801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:06.671654 9801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:06.671839 9801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:06.676116 9801 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:45815
I20260925 14:55:06.676149 9868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:45815 every 8 connection(s)
I20260925 14:55:06.676644 9801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:06.677505 9869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:06.677547 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 9801
I20260925 14:55:06.677678 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:06.678319 9781 external_mini_cluster.cc:1428] Setting key 2f2baa4d096cd541139eefd8875b8af1
I20260925 14:55:06.680272 9869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611: Bootstrap starting.
I20260925 14:55:06.680987 9869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:06.681298 9869 log.cc:824] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:06.682127 9869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611: No bootstrap required, opened a new log
I20260925 14:55:06.684043 9869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64587eaff546c0bc39ef2bded64611" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45815 } }
I20260925 14:55:06.684193 9869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:06.684219 9869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c64587eaff546c0bc39ef2bded64611, State: Initialized, Role: FOLLOWER
I20260925 14:55:06.684398 9869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64587eaff546c0bc39ef2bded64611" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45815 } }
I20260925 14:55:06.684489 9869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:06.684515 9869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:06.684546 9869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:06.685142 9869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64587eaff546c0bc39ef2bded64611" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45815 } }
I20260925 14:55:06.685236 9869 leader_election.cc:302] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c64587eaff546c0bc39ef2bded64611; no voters:
I20260925 14:55:06.685456 9869 leader_election.cc:288] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:06.685511 9874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:06.685648 9874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [term 1 LEADER]: Becoming Leader. State: Replica: 2c64587eaff546c0bc39ef2bded64611, State: Running, Role: LEADER
I20260925 14:55:06.685727 9874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64587eaff546c0bc39ef2bded64611" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45815 } }
I20260925 14:55:06.685781 9869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:06.686040 9874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c64587eaff546c0bc39ef2bded64611. Latest consensus state: current_term: 1 leader_uuid: "2c64587eaff546c0bc39ef2bded64611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64587eaff546c0bc39ef2bded64611" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45815 } } }
I20260925 14:55:06.686129 9874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:06.686049 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c64587eaff546c0bc39ef2bded64611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c64587eaff546c0bc39ef2bded64611" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45815 } } }
I20260925 14:55:06.686314 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:06.686354 9878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:06.686866 9878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:06.689168 9878 catalog_manager.cc:1347] Generated new cluster ID: e9021a59495c40b79ec40a105fc2c335
I20260925 14:55:06.689219 9878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:06 dist-test-slave-1sv7 krb5kdc[9788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:06.696516 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:06.679175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.9.141.126:45815)
0925 14:55:06.679353 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:06.679361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:06.679601 (+ 240us) server_negotiation.cc:407] Connection header received
0925 14:55:06.682653 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:06.682662 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:06.682729 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:06.682891 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:06.684359 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.685823 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.687363 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.687913 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.688035 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:06.692458 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:06.692463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:06.692463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:06.692480 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:06.694440 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.694964 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.694967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.694967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.695071 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.695361 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.695362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.695362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.695655 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:06.695807 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:06.696133 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 14:55:06.696266 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260925 14:55:06.716861 9878 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:06.717449 9878 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:06.725196 9878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c64587eaff546c0bc39ef2bded64611: Generated new TSK 0
I20260925 14:55:06.725400 9878 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:06.757596 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45815
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:06.839758 9895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:06.840010 9895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:06.840047 9895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:06.841362 9895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:06.841435 9895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:06.841488 9895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:06.842991 9895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:06.843325 9895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:06.843575 9895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:06.848742 9900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:06.848735 9903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:06.848910 9901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:06.849097 9895 server_base.cc:1034] running on GCE node
I20260925 14:55:06.849318 9895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:06.849500 9895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:06.850667 9895 hybrid_clock.cc:648] HybridClock initialized: now 1790348106850652 us; error 35 us; skew 500 ppm
Sep 25 14:55:06 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:06.853121 9895 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:06.853531 9895 webserver.cc:460] Webserver started at http://127.9.141.65:37957/ using document root <none> and password file <none>
I20260925 14:55:06.853755 9895 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:06.853824 9895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:06.853976 9895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:06.855022 9895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "dd30fcdd22fb4cd0a62103d1b50674e4"
format_stamp: "Formatted at 2026-09-25 14:55:06 on dist-test-slave-1sv7"
server_key: "400a4b394a2270761c6c73db814a2d3f"
server_key_iv: "1d93e6a8686352fe4d3760eafc2159cc"
server_key_version: "encryptionkey@0"
I20260925 14:55:06.855388 9895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "dd30fcdd22fb4cd0a62103d1b50674e4"
format_stamp: "Formatted at 2026-09-25 14:55:06 on dist-test-slave-1sv7"
server_key: "400a4b394a2270761c6c73db814a2d3f"
server_key_iv: "1d93e6a8686352fe4d3760eafc2159cc"
server_key_version: "encryptionkey@0"
I20260925 14:55:06.856956 9895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 14:55:06.857757 9910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:06.857954 9895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:06.858048 9895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "dd30fcdd22fb4cd0a62103d1b50674e4"
format_stamp: "Formatted at 2026-09-25 14:55:06 on dist-test-slave-1sv7"
server_key: "400a4b394a2270761c6c73db814a2d3f"
server_key_iv: "1d93e6a8686352fe4d3760eafc2159cc"
server_key_version: "encryptionkey@0"
I20260925 14:55:06.858134 9895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:06.876087 9895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:06.878639 9895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:06.878798 9895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:06.879037 9895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:06.879351 9895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:06.879405 9895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:06.879447 9895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:06.879474 9895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:06.885303 9895 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:41599
I20260925 14:55:06.885373 10023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:41599 every 8 connection(s)
I20260925 14:55:06.885700 9895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:55:06 dist-test-slave-1sv7 krb5kdc[9788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348106, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:06.892614 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 9895
I20260925 14:55:06.892692 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:06.892874 9781 external_mini_cluster.cc:1428] Setting key 6a20611360085a5c364659f1ab600715
I20260925 14:55:06.895810 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:06.886472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:33073 (local address 127.9.141.126:45815)
0925 14:55:06.886521 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:06.886524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:06.886754 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:55:06.887273 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:06.887275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:06.887287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:06.887352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:06.888313 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.888865 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.889865 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.890099 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.890171 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:06.892576 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:06.892577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:06.892577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:06.892584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:06.894130 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.894656 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.894657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.894657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.894708 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:06.895028 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:06.895029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:06.895029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:06.895190 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:06.895267 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:06.895592 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 14:55:06.895690 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:06.895848 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:06.886432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45815 (local address 127.9.141.65:33073)
0925 14:55:06.886621 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:06.886657 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:06.887164 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:06.887423 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:06.887427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:06.888009 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:06.888204 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:06.888208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.888959 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.888960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:06.889767 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:06.889769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:06.890189 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:06.890190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:06.890320 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:06.890784 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:06.890792 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:06.892453 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:06.894215 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:06.894221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:06.894224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:06.894579 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:06.894778 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:06.894779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:06.894780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:06.894945 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:06.895298 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:06.895304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:06.895444 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:55:06.895554 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:55:06.895684 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260925 14:55:06.896471 10024 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45815
I20260925 14:55:06.896566 10024 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:06.896759 10024 heartbeater.cc:511] Master 127.9.141.126:45815 requested a full tablet report, sending...
I20260925 14:55:06.897284 9833 ts_manager.cc:194] Registered new tserver with Master: dd30fcdd22fb4cd0a62103d1b50674e4 (127.9.141.65:41599)
I20260925 14:55:06.897898 9833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:33073
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:06.919898 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45815
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:07.000612 10029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:07.000806 10029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:07.000828 10029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:07.001878 10029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:07.001911 10029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:07.001937 10029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:07.003270 10029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:07.003446 10029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:07.003620 10029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:07.008793 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:07.008744 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:07.008744 10037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:07.008994 10029 server_base.cc:1034] running on GCE node
I20260925 14:55:07.009195 10029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:07.009438 10029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:07.010612 10029 hybrid_clock.cc:648] HybridClock initialized: now 1790348107010597 us; error 34 us; skew 500 ppm
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:07.013190 10029 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:07.013608 10029 webserver.cc:460] Webserver started at http://127.9.141.66:40549/ using document root <none> and password file <none>
I20260925 14:55:07.013880 10029 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:07.013947 10029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:07.014096 10029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:07.015112 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "b9c261eacb8847c7a3776f740f36d80d"
format_stamp: "Formatted at 2026-09-25 14:55:07 on dist-test-slave-1sv7"
server_key: "fd4d9a44b7231eab0bb4ebd29c4e899d"
server_key_iv: "f3da8ca4feabc9fc6619bceb9f23664b"
server_key_version: "encryptionkey@0"
I20260925 14:55:07.015486 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "b9c261eacb8847c7a3776f740f36d80d"
format_stamp: "Formatted at 2026-09-25 14:55:07 on dist-test-slave-1sv7"
server_key: "fd4d9a44b7231eab0bb4ebd29c4e899d"
server_key_iv: "f3da8ca4feabc9fc6619bceb9f23664b"
server_key_version: "encryptionkey@0"
I20260925 14:55:07.016855 10029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:07.017635 10044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.017863 10029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:07.017972 10029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "b9c261eacb8847c7a3776f740f36d80d"
format_stamp: "Formatted at 2026-09-25 14:55:07 on dist-test-slave-1sv7"
server_key: "fd4d9a44b7231eab0bb4ebd29c4e899d"
server_key_iv: "f3da8ca4feabc9fc6619bceb9f23664b"
server_key_version: "encryptionkey@0"
I20260925 14:55:07.018054 10029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:07.027093 10029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:07.029353 10029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:07.029523 10029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:07.029738 10029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:07.030040 10029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:07.030092 10029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.030123 10029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:07.030150 10029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.035735 10029 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:45775
I20260925 14:55:07.035808 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:45775 every 8 connection(s)
I20260925 14:55:07.036120 10029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:07.044850 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10029
I20260925 14:55:07.044967 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:07.045150 9781 external_mini_cluster.cc:1428] Setting key d767b06e9d093481219ec1f8b664a3b7
I20260925 14:55:07.045951 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.036862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:57157 (local address 127.9.141.126:45815)
0925 14:55:07.036909 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:07.036912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:07.037128 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:55:07.037632 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:07.037634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:07.037643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:07.037702 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:07.038615 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.039125 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.040158 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.040354 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.040428 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.042584 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:07.042586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:07.042586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:07.042592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:07.044052 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.044705 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.044707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.044707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.044784 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.045177 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.045178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.045179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.045352 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:07.045495 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:07.045766 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:55:07.045862 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:07.046119 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.036847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45815 (local address 127.9.141.66:57157)
0925 14:55:07.037012 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:07.037045 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:07.037488 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:07.037784 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:07.037787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:07.038326 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:07.038510 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:07.038514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.039231 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.039232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:07.040056 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:07.040058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.040512 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.040515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:07.040625 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.041087 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:07.041095 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:07.042467 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:07.044143 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:07.044148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:07.044149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:07.044353 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:07.044877 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:07.044878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:07.044878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:07.045089 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:07.045487 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:07.045493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:07.045620 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 14:55:07.045726 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:55:07.045878 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260925 14:55:07.046517 10158 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45815
I20260925 14:55:07.046605 10158 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:07.046787 10158 heartbeater.cc:511] Master 127.9.141.126:45815 requested a full tablet report, sending...
I20260925 14:55:07.047163 9833 ts_manager.cc:194] Registered new tserver with Master: b9c261eacb8847c7a3776f740f36d80d (127.9.141.66:45775)
I20260925 14:55:07.047663 9833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:57157
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:07.071837 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45815
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:07.150607 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:07.150831 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:07.150867 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:07.151994 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:07.152064 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:07.152107 10163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:07.153589 10163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:07.153856 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:07.154088 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:07.159416 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:07.159416 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:07.159443 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:07.159736 10163 server_base.cc:1034] running on GCE node
I20260925 14:55:07.159973 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:07.160169 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:07.161339 10163 hybrid_clock.cc:648] HybridClock initialized: now 1790348107161317 us; error 44 us; skew 500 ppm
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:07.164067 10163 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:07.164530 10163 webserver.cc:460] Webserver started at http://127.9.141.67:32857/ using document root <none> and password file <none>
I20260925 14:55:07.164770 10163 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:07.164815 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:07.164950 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:07.166090 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "33dafb2481b146f2a7df6154067c7ef9"
format_stamp: "Formatted at 2026-09-25 14:55:07 on dist-test-slave-1sv7"
server_key: "c294ef7e7a932156c34ca8ae18d0ab03"
server_key_iv: "7853ed322a8728e859d7f713274a6e66"
server_key_version: "encryptionkey@0"
I20260925 14:55:07.166432 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "33dafb2481b146f2a7df6154067c7ef9"
format_stamp: "Formatted at 2026-09-25 14:55:07 on dist-test-slave-1sv7"
server_key: "c294ef7e7a932156c34ca8ae18d0ab03"
server_key_iv: "7853ed322a8728e859d7f713274a6e66"
server_key_version: "encryptionkey@0"
I20260925 14:55:07.167733 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:07.168560 10178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.168740 10163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:07.168828 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "33dafb2481b146f2a7df6154067c7ef9"
format_stamp: "Formatted at 2026-09-25 14:55:07 on dist-test-slave-1sv7"
server_key: "c294ef7e7a932156c34ca8ae18d0ab03"
server_key_iv: "7853ed322a8728e859d7f713274a6e66"
server_key_version: "encryptionkey@0"
I20260925 14:55:07.168896 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:07.206636 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:07.209036 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:07.209198 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:07.209451 10163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:07.209769 10163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:07.209823 10163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.209864 10163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:07.209892 10163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:07.215623 10163 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:34785
I20260925 14:55:07.215664 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:34785 every 8 connection(s)
I20260925 14:55:07.216030 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:07.217092 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10163
I20260925 14:55:07.217197 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:07.217370 9781 external_mini_cluster.cc:1428] Setting key e8bec55450b90b7ce966828432fa8129
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:07.226405 9872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.216854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:47783 (local address 127.9.141.126:45815)
0925 14:55:07.216962 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:07.216964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:07.217130 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:55:07.217874 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:07.217875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:07.217884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:07.217999 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:07.219117 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.219656 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.220787 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.221014 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.221083 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.223203 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:07.223204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:07.223204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:07.223210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:07.224646 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.225061 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.225062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.225063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.225135 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.225741 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.225743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.225743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.225889 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:07.225954 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:07.226277 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 14:55:07.226342 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 14:55:07.226614 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.216826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45815 (local address 127.9.141.67:47783)
0925 14:55:07.216993 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:07.217033 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:07.217654 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:07.218084 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:07.218088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:07.218827 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:07.219025 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:07.219030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.219745 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.219746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:07.220686 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:07.220688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.221088 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.221089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:07.221187 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.221725 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:07.221734 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:07.223118 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:07.224738 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:07.224743 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:07.224745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:07.224985 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:07.225349 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:07.225350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:07.225351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:07.225557 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:07.226017 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:07.226023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:07.226148 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 14:55:07.226249 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:55:07.226373 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260925 14:55:07.227169 10292 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45815
I20260925 14:55:07.227258 10292 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:07.227391 10292 heartbeater.cc:511] Master 127.9.141.126:45815 requested a full tablet report, sending...
I20260925 14:55:07.227766 9833 ts_manager.cc:194] Registered new tserver with Master: 33dafb2481b146f2a7df6154067c7ef9 (127.9.141.67:34785)
I20260925 14:55:07.228101 9833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:47783
I20260925 14:55:07.229723 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:07.252936 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.245036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.9.141.65:41599)
0925 14:55:07.245263 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:07.245268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:07.245278 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:07.245304 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:07.245307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:07.245320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:07.245386 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:07.246027 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.246718 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.247519 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.247828 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.247883 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.250129 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:07.250139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:07.250139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:07.250146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:07.251786 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.252145 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.252147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.252148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.252219 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.252458 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.252459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.252459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.252600 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:07.252722 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:07.252786 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:55:07.252852 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":119,"threads_started":1}
W20260925 14:55:07.253350 9938 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43192
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[9788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:07.275682 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:07.268498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.9.141.65:41599)
0925 14:55:07.268571 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:07.268576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:07.268590 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:55:07.268614 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:07.268615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:07.268626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:07.268683 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:07.269274 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.269805 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.270573 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:07.270776 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:07.270840 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:07.273353 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:07.273354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:07.273354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:07.273362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:07.274633 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.274994 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.274995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.274995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.275053 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:07.275236 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:07.275237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:07.275237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:07.275333 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:07.275479 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:07.275543 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:55:07.275613 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:55:07.276325 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 9895
I20260925 14:55:07.280998 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10029
I20260925 14:55:07.286053 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10163
I20260925 14:55:07.290196 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 9801
2026-09-25T14:55:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1934 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[10306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[10306](info): set up 2 sockets
Sep 25 14:55:07 dist-test-slave-1sv7 krb5kdc[10306](info): commencing operation
krb5kdc: starting...
W20260925 14:55:08.347837 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:08.467811 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46841
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:46841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:08.548354 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:08.548650 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:08.548672 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:08.549782 10319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:08.549817 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:08.549831 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:08.549844 10319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:08.549855 10319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:08.551226 10319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:46841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:08.551424 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:08.551595 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:08.556599 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.556586 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.556589 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:08.556831 10319 server_base.cc:1034] running on GCE node
I20260925 14:55:08.557080 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:08.557365 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:08.558512 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790348108558495 us; error 40 us; skew 500 ppm
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:08.561093 10319 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:08.561594 10319 webserver.cc:460] Webserver started at http://127.9.141.126:46439/ using document root <none> and password file <none>
I20260925 14:55:08.561841 10319 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:08.561911 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:08.562063 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:08.563115 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "f8084ba635c344279be21e796c107fd5"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "923f95d3ca3f1435975c5759b0b82cd0"
server_key_iv: "c9a4f06fce34fc837d12a35c68461cfb"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.563473 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "f8084ba635c344279be21e796c107fd5"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "923f95d3ca3f1435975c5759b0b82cd0"
server_key_iv: "c9a4f06fce34fc837d12a35c68461cfb"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.564894 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:08.565657 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.565883 10319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:08.565982 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "f8084ba635c344279be21e796c107fd5"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "923f95d3ca3f1435975c5759b0b82cd0"
server_key_iv: "c9a4f06fce34fc837d12a35c68461cfb"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.566059 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:08.598250 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:08.600477 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:08.600643 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:08.605168 10319 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:46841
I20260925 14:55:08.605223 10386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:46841 every 8 connection(s)
I20260925 14:55:08.605638 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:08.606539 10387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:08.608955 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5: Bootstrap starting.
I20260925 14:55:08.609552 10387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:08.609870 10387 log.cc:824] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:08.610757 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5: No bootstrap required, opened a new log
I20260925 14:55:08.612504 10387 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8084ba635c344279be21e796c107fd5" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46841 } }
I20260925 14:55:08.612654 10387 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:08.612676 10387 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8084ba635c344279be21e796c107fd5, State: Initialized, Role: FOLLOWER
I20260925 14:55:08.612823 10387 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8084ba635c344279be21e796c107fd5" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46841 } }
I20260925 14:55:08.612900 10387 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:08.612946 10387 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:08.612995 10387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:08.613216 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10319
I20260925 14:55:08.613320 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:08.613503 9781 external_mini_cluster.cc:1428] Setting key b815bff9e0153e1fbd767d739a9206fa
I20260925 14:55:08.613602 10387 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8084ba635c344279be21e796c107fd5" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46841 } }
I20260925 14:55:08.613729 10387 leader_election.cc:302] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8084ba635c344279be21e796c107fd5; no voters:
I20260925 14:55:08.613897 10387 leader_election.cc:288] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:08.613938 10390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:08.614099 10390 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [term 1 LEADER]: Becoming Leader. State: Replica: f8084ba635c344279be21e796c107fd5, State: Running, Role: LEADER
I20260925 14:55:08.614152 10387 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:08.614202 10390 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8084ba635c344279be21e796c107fd5" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46841 } }
I20260925 14:55:08.614538 10390 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8084ba635c344279be21e796c107fd5. Latest consensus state: current_term: 1 leader_uuid: "f8084ba635c344279be21e796c107fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8084ba635c344279be21e796c107fd5" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46841 } } }
I20260925 14:55:08.614637 10390 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:08.614557 10392 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8084ba635c344279be21e796c107fd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8084ba635c344279be21e796c107fd5" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46841 } } }
I20260925 14:55:08.614778 10392 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:08.615432 10396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:08.615974 10396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:08.618225 10396 catalog_manager.cc:1347] Generated new cluster ID: fe3ab706cc1542f7ab19aa174a23a4af
I20260925 14:55:08.618276 10396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:08.625675 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.614146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.9.141.126:46841)
0925 14:55:08.614895 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:08.614906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:08.614951 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:55:08.615076 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:08.615083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:08.615142 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:08.615303 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:08.616094 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.617501 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.618569 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.619188 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.619301 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.622077 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:08.622081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:08.622081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:08.622093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:08.624043 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.624457 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.624460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.624461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.624577 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.624788 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.624789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.624789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.625067 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:08.625212 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:08.625342 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:55:08.625475 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":68,"threads_started":1}
I20260925 14:55:08.636482 10396 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:08.637236 10396 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:08.647568 10396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8084ba635c344279be21e796c107fd5: Generated new TSK 0
I20260925 14:55:08.647783 10396 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:08.679982 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46841
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:08.764233 10413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:08.764489 10413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:08.764516 10413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:08.765645 10413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:08.765682 10413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:08.765708 10413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:08.767145 10413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:08.767395 10413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:08.767617 10413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:08.772671 10419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.772698 10418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.772697 10421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:08.772806 10413 server_base.cc:1034] running on GCE node
I20260925 14:55:08.772985 10413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:08.773228 10413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:08.774364 10413 hybrid_clock.cc:648] HybridClock initialized: now 1790348108774340 us; error 44 us; skew 500 ppm
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:08.776777 10413 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:08.777153 10413 webserver.cc:460] Webserver started at http://127.9.141.65:42999/ using document root <none> and password file <none>
I20260925 14:55:08.777398 10413 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:08.777447 10413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:08.777588 10413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:08.778640 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "2a092513ad204190beb11c52156fd89c"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "9d0b0206667cd0cbffa25e9e5c27caf3"
server_key_iv: "d7db768507db2c90ef6090f45a02fd85"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.778997 10413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "2a092513ad204190beb11c52156fd89c"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "9d0b0206667cd0cbffa25e9e5c27caf3"
server_key_iv: "d7db768507db2c90ef6090f45a02fd85"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.780347 10413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:08.781497 10428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.781677 10413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:08.781764 10413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "2a092513ad204190beb11c52156fd89c"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "9d0b0206667cd0cbffa25e9e5c27caf3"
server_key_iv: "d7db768507db2c90ef6090f45a02fd85"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.781821 10413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:08.797111 10413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:08.799145 10413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:08.799258 10413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:08.799449 10413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:08.799751 10413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:08.799784 10413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.799808 10413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:08.799855 10413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.806286 10413 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:39511
I20260925 14:55:08.806342 10541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:39511 every 8 connection(s)
I20260925 14:55:08.806682 10413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:08.814855 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10413
I20260925 14:55:08.814949 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:08.815125 9781 external_mini_cluster.cc:1428] Setting key b721282c4c56fae1d58874b4760de0d9
I20260925 14:55:08.817078 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.807522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46841 (local address 127.9.141.65:56999)
0925 14:55:08.807705 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:08.807746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:08.808695 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:08.808991 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:08.808995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:08.809245 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:08.809495 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:08.809499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.810162 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.810163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:08.811332 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:08.811335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.811748 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.811749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:08.811880 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.812445 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:08.812454 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:08.813843 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:08.815624 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:08.815630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:08.815633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:08.815938 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:08.816168 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:08.816169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:08.816169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:08.816284 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:08.816592 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:08.816597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:08.816707 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 14:55:08.816793 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:55:08.816919 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260925 14:55:08.817091 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.807587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:56999 (local address 127.9.141.126:46841)
0925 14:55:08.807648 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:08.807650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:08.807844 (+ 194us) server_negotiation.cc:407] Connection header received
0925 14:55:08.808826 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:08.808828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:08.808845 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:08.808914 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:08.809584 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.810083 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.811433 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.811645 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.811737 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.813940 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:08.813941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:08.813941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:08.813950 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:08.815554 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.816024 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.816025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.816026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.816089 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.816352 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.816353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.816354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.816525 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:08.816597 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:08.816837 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:55:08.816950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:55:08.817822 10542 heartbeater.cc:347] Connected to a master server at 127.9.141.126:46841
I20260925 14:55:08.817937 10542 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:08.818127 10542 heartbeater.cc:511] Master 127.9.141.126:46841 requested a full tablet report, sending...
I20260925 14:55:08.818634 10351 ts_manager.cc:194] Registered new tserver with Master: 2a092513ad204190beb11c52156fd89c (127.9.141.65:39511)
I20260925 14:55:08.819236 10351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:56999
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:08.843793 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46841
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:08.926390 10547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:08.926579 10547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:08.926602 10547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:08.927666 10547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:08.927698 10547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:08.927723 10547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:08.929186 10547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:08.929380 10547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:08.929553 10547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:08.934617 10553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.934592 10555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:08.934592 10552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:08.935056 10547 server_base.cc:1034] running on GCE node
I20260925 14:55:08.935408 10547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:08.935688 10547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:08.936894 10547 hybrid_clock.cc:648] HybridClock initialized: now 1790348108936888 us; error 91 us; skew 500 ppm
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:08.946048 10547 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:08.946539 10547 webserver.cc:460] Webserver started at http://127.9.141.66:34439/ using document root <none> and password file <none>
I20260925 14:55:08.946791 10547 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:08.946868 10547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:08.947016 10547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:08.948101 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "3699d986fadb49fd894cb2f6d3df2d1e"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "df5aafa1047b28af50758f18b7699e35"
server_key_iv: "152322c59f672e401d3b3e947fa249c5"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.948536 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "3699d986fadb49fd894cb2f6d3df2d1e"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "df5aafa1047b28af50758f18b7699e35"
server_key_iv: "152322c59f672e401d3b3e947fa249c5"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.949867 10547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:08.950572 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.950738 10547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:08.950837 10547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "3699d986fadb49fd894cb2f6d3df2d1e"
format_stamp: "Formatted at 2026-09-25 14:55:08 on dist-test-slave-1sv7"
server_key: "df5aafa1047b28af50758f18b7699e35"
server_key_iv: "152322c59f672e401d3b3e947fa249c5"
server_key_version: "encryptionkey@0"
I20260925 14:55:08.950923 10547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:08.969254 10547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:08.971705 10547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:08.971884 10547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:08.972123 10547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:08.972498 10547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:08.972550 10547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.972590 10547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:08.972628 10547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:08.978623 10547 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:35053
I20260925 14:55:08.978667 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:35053 every 8 connection(s)
I20260925 14:55:08.979043 10547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:55:08 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:08.989012 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.979789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:48899 (local address 127.9.141.126:46841)
0925 14:55:08.979838 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:08.979840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:08.980111 (+ 271us) server_negotiation.cc:407] Connection header received
0925 14:55:08.981081 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:08.981083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:08.981098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:08.981154 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:08.981774 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.982268 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.983255 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.983490 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.983552 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.985891 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:08.985892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:08.985892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:08.985898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:08.987430 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.987851 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.987852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.987853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.987906 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:08.988190 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:08.988191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:08.988192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:08.988356 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:08.988501 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:08.988811 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 14:55:08.988920 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:08.989145 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10547
I20260925 14:55:08.989200 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:08.979780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46841 (local address 127.9.141.66:48899)
0925 14:55:08.979974 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:08.980008 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:08.980982 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:08.981223 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:08.981226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:08.981505 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:08.981691 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:08.981695 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.982375 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.982376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:08.983175 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:08.983177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:08.983597 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:08.983598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:08.983710 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:08.984255 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:08.984264 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:08.985806 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:08.987541 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:08.987547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:08.987548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:08.987769 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:08.987993 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:08.987994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:08.987995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:08.988101 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:08.988512 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:08.988518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:08.988638 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:55:08.988774 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:55:08.988960 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260925 14:55:08.989310 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:08.989493 9781 external_mini_cluster.cc:1428] Setting key f570858b2e5102857a5fa5329d43b41f
I20260925 14:55:08.989703 10676 heartbeater.cc:347] Connected to a master server at 127.9.141.126:46841
I20260925 14:55:08.989806 10676 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:08.989995 10676 heartbeater.cc:511] Master 127.9.141.126:46841 requested a full tablet report, sending...
I20260925 14:55:08.990365 10351 ts_manager.cc:194] Registered new tserver with Master: 3699d986fadb49fd894cb2f6d3df2d1e (127.9.141.66:35053)
I20260925 14:55:08.990757 10351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:48899
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:09.015596 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46841
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:09.095865 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:09.096068 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:09.096091 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:09.097185 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:09.097218 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:09.097244 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:09.098625 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:09.098801 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:09.098968 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:09.103884 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:09.103929 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:09.103900 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:09.104130 10681 server_base.cc:1034] running on GCE node
I20260925 14:55:09.104439 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:09.104707 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:09.105891 10681 hybrid_clock.cc:648] HybridClock initialized: now 1790348109105885 us; error 35 us; skew 500 ppm
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:09.108611 10681 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:09.109017 10681 webserver.cc:460] Webserver started at http://127.9.141.67:34995/ using document root <none> and password file <none>
I20260925 14:55:09.109249 10681 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:09.109323 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:09.109480 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:09.110548 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "9a150eedf2274774a1b34daa58c5bbd7"
format_stamp: "Formatted at 2026-09-25 14:55:09 on dist-test-slave-1sv7"
server_key: "4abf245dd583f906fab6f5e73ccb4e2f"
server_key_iv: "4b509587676186f36017d1d20776995a"
server_key_version: "encryptionkey@0"
I20260925 14:55:09.110944 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "9a150eedf2274774a1b34daa58c5bbd7"
format_stamp: "Formatted at 2026-09-25 14:55:09 on dist-test-slave-1sv7"
server_key: "4abf245dd583f906fab6f5e73ccb4e2f"
server_key_iv: "4b509587676186f36017d1d20776995a"
server_key_version: "encryptionkey@0"
I20260925 14:55:09.112272 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:09.113106 10696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:09.113310 10681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:09.113406 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "9a150eedf2274774a1b34daa58c5bbd7"
format_stamp: "Formatted at 2026-09-25 14:55:09 on dist-test-slave-1sv7"
server_key: "4abf245dd583f906fab6f5e73ccb4e2f"
server_key_iv: "4b509587676186f36017d1d20776995a"
server_key_version: "encryptionkey@0"
I20260925 14:55:09.113485 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:09.134896 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:09.137418 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:09.137594 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:09.137842 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:09.138164 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:09.138217 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:09.138252 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:09.138283 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:09.144145 10681 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:42121
I20260925 14:55:09.144249 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:42121 every 8 connection(s)
I20260925 14:55:09.144588 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:09.150692 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10681
I20260925 14:55:09.150908 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:09.151079 9781 external_mini_cluster.cc:1428] Setting key 60950e77ffa9d32cd09cdfcd16e16405
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:09.155159 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.145328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:34553 (local address 127.9.141.126:46841)
0925 14:55:09.145374 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.145377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.145599 (+ 222us) server_negotiation.cc:407] Connection header received
0925 14:55:09.146576 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.146577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.146591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.146642 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.147314 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.147959 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.149326 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.149515 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.149580 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.151993 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.151994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.151994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.152002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.153503 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.153956 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.153957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.153958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.154016 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.154281 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.154282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.154283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.154540 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.154654 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.154970 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.155056 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:09.155350 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.145317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46841 (local address 127.9.141.67:34553)
0925 14:55:09.145472 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.145507 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.146454 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.146715 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.146719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.147013 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:09.147210 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.147215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.148107 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.148108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.149230 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.149233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.149615 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.149616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.149731 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.150231 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:09.150239 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:09.151907 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:09.153607 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:09.153613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:09.153615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:09.153861 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:09.154094 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:09.154095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:09.154095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:09.154211 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:09.154644 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:09.154650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:09.154820 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.154936 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.155098 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260925 14:55:09.155921 10810 heartbeater.cc:347] Connected to a master server at 127.9.141.126:46841
I20260925 14:55:09.156006 10810 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:09.156219 10810 heartbeater.cc:511] Master 127.9.141.126:46841 requested a full tablet report, sending...
I20260925 14:55:09.156744 10351 ts_manager.cc:194] Registered new tserver with Master: 9a150eedf2274774a1b34daa58c5bbd7 (127.9.141.67:42121)
I20260925 14:55:09.157126 10351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:34553
I20260925 14:55:09.163066 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:09.170672 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.164682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.9.141.126:46841)
0925 14:55:09.164745 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.164747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.164888 (+ 141us) server_negotiation.cc:407] Connection header received
0925 14:55:09.164914 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.164914 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.164923 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.164974 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.165551 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.166083 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.166784 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.166967 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.167026 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.168255 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.168256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.168256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.168263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.169636 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.169983 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.169984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.169984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.170035 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.170175 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.170176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.170176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.170301 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.170372 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.170513 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.170573 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:09.172665 10351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:09.174180 10351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:09.180683 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.176801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:35053 (local address 127.0.0.1:59788)
0925 14:55:09.177298 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.177307 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.177354 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.177547 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.177548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.177550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.177641 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.177644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.178594 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.178595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.179432 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.179434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.180393 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.180394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.180470 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.180485 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.180558 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.180607 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":106,"threads_started":1}
I20260925 14:55:09.180702 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.176975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.9.141.66:35053)
0925 14:55:09.177156 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.177162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.177404 (+ 242us) server_negotiation.cc:407] Connection header received
0925 14:55:09.177450 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.177451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.177462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.177551 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.177724 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.178490 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.179541 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.180291 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.180408 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.180470 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.180569 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.180607 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260925 14:55:09.181103 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.176792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:42121 (local address 127.0.0.1:60958)
0925 14:55:09.176982 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.177024 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.177088 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.177315 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.177318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.177320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.177387 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.177389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.178648 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.178649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.179823 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.179825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.180835 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.180836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.180909 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.180913 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.180989 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.181030 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":42,"thread_start_us":96,"threads_started":1}
I20260925 14:55:09.181118 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.176965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.9.141.67:42121)
0925 14:55:09.177137 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.177143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.177152 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:09.177189 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.177191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.177205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.177302 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.177447 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.178532 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.179909 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.180641 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.180737 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.180800 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.181004 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.181043 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260925 14:55:09.181576 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.176792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39511 (local address 127.0.0.1:40332)
0925 14:55:09.177225 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.177235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.177295 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.177869 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.177870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.177872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.177906 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.177907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.179162 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.179171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.180193 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.180195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.181220 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.181221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.181343 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.181353 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.181436 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.181492 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":2,"thread_start_us":42,"threads_started":1}
I20260925 14:55:09.181887 10744 tablet_service.cc:1467] Processing CreateTablet for tablet d74895eb16084b2391044021348f441c (DEFAULT_TABLE table=test-table [id=0bcea2e67e814a798bf75087a36192cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:09.181888 10610 tablet_service.cc:1467] Processing CreateTablet for tablet d74895eb16084b2391044021348f441c (DEFAULT_TABLE table=test-table [id=0bcea2e67e814a798bf75087a36192cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:09.182248 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74895eb16084b2391044021348f441c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:09.182247 10744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74895eb16084b2391044021348f441c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:09.184662 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.177020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.9.141.65:39511)
0925 14:55:09.177177 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.177182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.177632 (+ 450us) server_negotiation.cc:407] Connection header received
0925 14:55:09.177680 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.177682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.177713 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.177788 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.177987 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.179057 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.180275 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.181036 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.181122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.181181 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.184476 (+ 3295us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.184548 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260925 14:55:09.185380 10476 tablet_service.cc:1467] Processing CreateTablet for tablet d74895eb16084b2391044021348f441c (DEFAULT_TABLE table=test-table [id=0bcea2e67e814a798bf75087a36192cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:09.185747 10476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d74895eb16084b2391044021348f441c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:09.186021 10828 tablet_bootstrap.cc:492] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: Bootstrap starting.
I20260925 14:55:09.186712 10828 tablet_bootstrap.cc:654] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:09.187322 10828 log.cc:824] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:09.185833 10827 tablet_bootstrap.cc:492] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: Bootstrap starting.
I20260925 14:55:09.188184 10828 tablet_bootstrap.cc:492] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: No bootstrap required, opened a new log
I20260925 14:55:09.188220 10827 tablet_bootstrap.cc:654] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:09.188279 10828 ts_tablet_manager.cc:1397] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:09.188809 10827 log.cc:824] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:09.189697 10827 tablet_bootstrap.cc:492] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: No bootstrap required, opened a new log
I20260925 14:55:09.189757 10827 ts_tablet_manager.cc:1397] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 14:55:09.190272 10828 raft_consensus.cc:348] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.190464 10828 raft_consensus.cc:374] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:09.190514 10828 raft_consensus.cc:729] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3699d986fadb49fd894cb2f6d3df2d1e, State: Initialized, Role: FOLLOWER
I20260925 14:55:09.190639 10828 consensus_queue.cc:260] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.190971 10828 ts_tablet_manager.cc:1428] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:09.190994 10676 heartbeater.cc:503] Master 127.9.141.126:46841 was elected leader, sending a full tablet report...
I20260925 14:55:09.191715 10827 raft_consensus.cc:348] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.191892 10827 raft_consensus.cc:374] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:09.191927 10827 raft_consensus.cc:729] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a150eedf2274774a1b34daa58c5bbd7, State: Initialized, Role: FOLLOWER
I20260925 14:55:09.192042 10827 consensus_queue.cc:260] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.192317 10810 heartbeater.cc:503] Master 127.9.141.126:46841 was elected leader, sending a full tablet report...
I20260925 14:55:09.192330 10827 ts_tablet_manager.cc:1428] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:55:09.192497 10831 tablet_bootstrap.cc:492] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: Bootstrap starting.
I20260925 14:55:09.193156 10831 tablet_bootstrap.cc:654] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:09.193645 10831 log.cc:824] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:09.194350 10831 tablet_bootstrap.cc:492] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: No bootstrap required, opened a new log
I20260925 14:55:09.194407 10831 ts_tablet_manager.cc:1397] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:09.195716 10831 raft_consensus.cc:348] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.195849 10831 raft_consensus.cc:374] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:09.195873 10831 raft_consensus.cc:729] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a092513ad204190beb11c52156fd89c, State: Initialized, Role: FOLLOWER
I20260925 14:55:09.195986 10831 consensus_queue.cc:260] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.196269 10831 ts_tablet_manager.cc:1428] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:09.196350 10542 heartbeater.cc:503] Master 127.9.141.126:46841 was elected leader, sending a full tablet report...
W20260925 14:55:09.229632 10677 tablet.cc:2369] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:09.307466 10543 tablet.cc:2369] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:09.335229 10832 raft_consensus.cc:482] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:09.335348 10832 raft_consensus.cc:504] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.335783 10832 leader_election.cc:288] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a092513ad204190beb11c52156fd89c (127.9.141.65:39511), 9a150eedf2274774a1b34daa58c5bbd7 (127.9.141.67:42121)
I20260925 14:55:09.338858 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.336004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:42121 (local address 127.9.141.66:60681)
0925 14:55:09.336043 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.336055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.336119 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.336284 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.336285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.336288 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.336369 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.336371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.337170 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.337171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.337983 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.337986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.338589 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.338590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.338670 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.338674 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.338738 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.338781 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:55:09.338873 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.336094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:60681 (local address 127.9.141.67:42121)
0925 14:55:09.336147 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.336151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.336165 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:55:09.336191 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.336191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.336201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.336257 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.336505 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.337086 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.338068 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.338495 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.338551 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.338558 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.338759 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.338800 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:55:09.339089 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.336080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:38917 (local address 127.9.141.65:39511)
0925 14:55:09.336153 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.336155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.336485 (+ 330us) server_negotiation.cc:407] Connection header received
0925 14:55:09.336537 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.336538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.336547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.336600 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:09.336746 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.337255 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.338211 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.338669 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.338747 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.338755 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.338975 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.339023 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:55:09.339238 10764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74895eb16084b2391044021348f441c" candidate_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a150eedf2274774a1b34daa58c5bbd7" is_pre_election: true
I20260925 14:55:09.339318 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.336004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39511 (local address 127.9.141.66:38917)
0925 14:55:09.336408 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:09.336419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:09.336480 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:09.336618 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:09.336619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:09.336621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:09.336681 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.336683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.337347 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.337348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.338106 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:09.338108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.338795 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.338796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:09.338868 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.338872 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:09.339170 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 14:55:09.339221 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":168,"thread_start_us":124,"threads_started":1}
I20260925 14:55:09.339432 10764 raft_consensus.cc:2454] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3699d986fadb49fd894cb2f6d3df2d1e in term 0.
I20260925 14:55:09.339686 10565 leader_election.cc:302] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3699d986fadb49fd894cb2f6d3df2d1e, 9a150eedf2274774a1b34daa58c5bbd7; no voters:
I20260925 14:55:09.339668 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74895eb16084b2391044021348f441c" candidate_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a092513ad204190beb11c52156fd89c" is_pre_election: true
I20260925 14:55:09.339823 10496 raft_consensus.cc:2454] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3699d986fadb49fd894cb2f6d3df2d1e in term 0.
I20260925 14:55:09.339823 10832 raft_consensus.cc:2781] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:09.339870 10832 raft_consensus.cc:482] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:09.339905 10832 raft_consensus.cc:3037] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:09.340667 10832 raft_consensus.cc:504] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.340816 10832 leader_election.cc:288] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [CANDIDATE]: Term 1 election: Requested vote from peers 2a092513ad204190beb11c52156fd89c (127.9.141.65:39511), 9a150eedf2274774a1b34daa58c5bbd7 (127.9.141.67:42121)
I20260925 14:55:09.340971 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74895eb16084b2391044021348f441c" candidate_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a092513ad204190beb11c52156fd89c"
I20260925 14:55:09.341003 10764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d74895eb16084b2391044021348f441c" candidate_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a150eedf2274774a1b34daa58c5bbd7"
I20260925 14:55:09.341051 10496 raft_consensus.cc:3037] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:09.341065 10764 raft_consensus.cc:3037] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:09.341878 10496 raft_consensus.cc:2454] T d74895eb16084b2391044021348f441c P 2a092513ad204190beb11c52156fd89c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3699d986fadb49fd894cb2f6d3df2d1e in term 1.
I20260925 14:55:09.341878 10764 raft_consensus.cc:2454] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3699d986fadb49fd894cb2f6d3df2d1e in term 1.
I20260925 14:55:09.342161 10565 leader_election.cc:302] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3699d986fadb49fd894cb2f6d3df2d1e, 9a150eedf2274774a1b34daa58c5bbd7; no voters:
I20260925 14:55:09.342253 10832 raft_consensus.cc:2781] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:09.342403 10832 raft_consensus.cc:686] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [term 1 LEADER]: Becoming Leader. State: Replica: 3699d986fadb49fd894cb2f6d3df2d1e, State: Running, Role: LEADER
I20260925 14:55:09.342504 10832 consensus_queue.cc:237] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } }
I20260925 14:55:09.343276 10351 catalog_manager.cc:5495] T d74895eb16084b2391044021348f441c P 3699d986fadb49fd894cb2f6d3df2d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3699d986fadb49fd894cb2f6d3df2d1e (127.9.141.66). New cstate: current_term: 1 leader_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a092513ad204190beb11c52156fd89c" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a150eedf2274774a1b34daa58c5bbd7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 42121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3699d986fadb49fd894cb2f6d3df2d1e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35053 } health_report { overall_health: HEALTHY } } }
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:09.371744 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.363469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.9.141.65:39511)
0925 14:55:09.363537 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.363539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.363717 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:55:09.363745 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.363754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.363764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.363832 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.364488 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.364983 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.365735 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.365921 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.365985 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.368246 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.368258 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.368258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.368264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.370422 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.370829 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.370834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.370836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.370905 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.371130 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.371132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.371132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.371295 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.371459 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.371531 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.371618 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:09.395131 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.388114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40346 (local address 127.9.141.65:39511)
0925 14:55:09.388224 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.388226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.388236 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:09.388283 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.388284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.388293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.388344 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.388970 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.389450 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.390245 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.390443 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.390504 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.392782 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.392783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.392783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.392790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.394141 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.394456 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.394456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.394457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.394504 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.394650 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.394651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.394651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.394740 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.394807 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.394943 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.394999 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 14:55:09.395196 10811 tablet.cc:2369] T d74895eb16084b2391044021348f441c P 9a150eedf2274774a1b34daa58c5bbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:09.417444 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:09.409824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.9.141.65:39511)
0925 14:55:09.409889 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:09.409891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:09.409952 (+ 61us) server_negotiation.cc:407] Connection header received
0925 14:55:09.410007 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:09.410008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:09.410016 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:09.410067 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:09.410665 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.411135 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.411977 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:09.412154 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:09.412212 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:09.414755 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:09.414757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:09.414758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:09.414765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:09.416196 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.416633 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.416634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.416635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.416690 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:09.416890 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:09.416891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:09.416891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:09.417046 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:09.417159 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:09.417262 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:55:09.417338 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 14:55:09.417900 10456 tablet_service.cc:2908] Rejecting scan request for tablet d74895eb16084b2391044021348f441c: Uninitialized: clean time has not yet been initialized
I20260925 14:55:09.418892 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10413
I20260925 14:55:09.423698 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10547
I20260925 14:55:09.428596 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10681
I20260925 14:55:09.433707 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10319
2026-09-25T14:55:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2143 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10851](info): set up 2 sockets
Sep 25 14:55:09 dist-test-slave-1sv7 krb5kdc[10851](info): commencing operation
krb5kdc: starting...
W20260925 14:55:10.503955 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:10 dist-test-slave-1sv7 krb5kdc[10851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:10.624033 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:40711
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:40711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:10.706168 10864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:10.706378 10864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:10.706400 10864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:10.707554 10864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:10.707597 10864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:55:10.707610 10864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:10.707623 10864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:10.707635 10864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:10.707648 10864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:10.709146 10864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:40711
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:10.709384 10864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:10.709633 10864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:10.714720 10869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:10.714752 10872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:10.715080 10870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:10.715553 10864 server_base.cc:1034] running on GCE node
I20260925 14:55:10.715849 10864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:10.716154 10864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:10.717347 10864 hybrid_clock.cc:648] HybridClock initialized: now 1790348110717323 us; error 70 us; skew 500 ppm
Sep 25 14:55:10 dist-test-slave-1sv7 krb5kdc[10851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348110, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:10.720170 10864 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:10.720662 10864 webserver.cc:460] Webserver started at http://127.9.141.126:46821/ using document root <none> and password file <none>
I20260925 14:55:10.720902 10864 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:10.720948 10864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:10.721101 10864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:10.722242 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "7dce37ecfdd84491b64f098844a5e491"
format_stamp: "Formatted at 2026-09-25 14:55:10 on dist-test-slave-1sv7"
server_key: "9eacd3e87a3695a7db47b760f25f25cc"
server_key_iv: "a354eb7bca2e44adb795470358f478db"
server_key_version: "encryptionkey@0"
I20260925 14:55:10.722604 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "7dce37ecfdd84491b64f098844a5e491"
format_stamp: "Formatted at 2026-09-25 14:55:10 on dist-test-slave-1sv7"
server_key: "9eacd3e87a3695a7db47b760f25f25cc"
server_key_iv: "a354eb7bca2e44adb795470358f478db"
server_key_version: "encryptionkey@0"
I20260925 14:55:10.724009 10864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:10.724898 10879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:10.725111 10864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:10.725231 10864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "7dce37ecfdd84491b64f098844a5e491"
format_stamp: "Formatted at 2026-09-25 14:55:10 on dist-test-slave-1sv7"
server_key: "9eacd3e87a3695a7db47b760f25f25cc"
server_key_iv: "a354eb7bca2e44adb795470358f478db"
server_key_version: "encryptionkey@0"
I20260925 14:55:10.725288 10864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:10.751915 10864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:10.754400 10864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:10.754518 10864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:10.759516 10864 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:40711
I20260925 14:55:10.759609 10941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:40711 every 8 connection(s)
I20260925 14:55:10.759944 10864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:10.760811 10942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:10.763341 10942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: Bootstrap starting.
I20260925 14:55:10.763935 10942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:10.764230 10942 log.cc:824] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:10.764914 10942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: No bootstrap required, opened a new log
I20260925 14:55:10.767006 10942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dce37ecfdd84491b64f098844a5e491" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40711 } }
I20260925 14:55:10.767184 10942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:10.767235 10942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dce37ecfdd84491b64f098844a5e491, State: Initialized, Role: FOLLOWER
I20260925 14:55:10.767383 10942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dce37ecfdd84491b64f098844a5e491" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40711 } }
I20260925 14:55:10.767463 10942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:10.767506 10942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:10.767551 10942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:10.768052 10942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dce37ecfdd84491b64f098844a5e491" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40711 } }
I20260925 14:55:10.768177 10942 leader_election.cc:302] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dce37ecfdd84491b64f098844a5e491; no voters:
I20260925 14:55:10.768352 10942 leader_election.cc:288] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:10.768468 10945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:10.768630 10942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:10.768656 10945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [term 1 LEADER]: Becoming Leader. State: Replica: 7dce37ecfdd84491b64f098844a5e491, State: Running, Role: LEADER
I20260925 14:55:10.768764 10945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dce37ecfdd84491b64f098844a5e491" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40711 } }
I20260925 14:55:10.769131 10946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dce37ecfdd84491b64f098844a5e491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dce37ecfdd84491b64f098844a5e491" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40711 } } }
I20260925 14:55:10.769213 10946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:10.769253 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10864
I20260925 14:55:10.769352 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:10.769522 9781 external_mini_cluster.cc:1428] Setting key b486f9c2501cbf8df16d9d4ad8750fe6
I20260925 14:55:10.770046 10947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dce37ecfdd84491b64f098844a5e491. Latest consensus state: current_term: 1 leader_uuid: "7dce37ecfdd84491b64f098844a5e491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dce37ecfdd84491b64f098844a5e491" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40711 } } }
I20260925 14:55:10.770310 10947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491 [sys.catalog]: This master's current role is: LEADER
W20260925 14:55:10.770438 10960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:10.770488 10960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:10.770830 10954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:10.771018 10954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:10.772351 10954 catalog_manager.cc:1347] Generated new cluster ID: 63d9ae5762ad4660ad711f0bc67a403a
I20260925 14:55:10.772421 10954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:10 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:10.782043 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.770008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.9.141.126:40711)
0925 14:55:10.770282 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:10.770292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:10.770548 (+ 256us) server_negotiation.cc:407] Connection header received
0925 14:55:10.770669 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:10.770678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:10.770738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:10.770866 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:10.771617 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.773049 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.774139 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.774755 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.774853 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.777944 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:10.777948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:10.777948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:10.777960 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:10.779909 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.780350 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.780353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.780354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.780504 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.780717 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.780718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.780718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.781029 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:10.781191 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:10.781671 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 14:55:10.781812 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
I20260925 14:55:10.792171 10954 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:10.792910 10954 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:10.801088 10954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: Generated new TSK 0
I20260925 14:55:10.801301 10954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:10.833282 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40711
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:10.916306 10968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:10.916572 10968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:10.916599 10968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:10.917788 10968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:10.917824 10968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:10.917845 10968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:10.917862 10968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:10.919209 10968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:40711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:10.919381 10968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:10.919546 10968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:10.924584 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:10.924577 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:10.924803 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:10.924935 10968 server_base.cc:1034] running on GCE node
I20260925 14:55:10.925184 10968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:10.925423 10968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:10.926584 10968 hybrid_clock.cc:648] HybridClock initialized: now 1790348110926572 us; error 29 us; skew 500 ppm
Sep 25 14:55:10 dist-test-slave-1sv7 krb5kdc[10851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348110, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:10.929293 10968 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:10.929687 10968 webserver.cc:460] Webserver started at http://127.9.141.65:35367/ using document root <none> and password file <none>
I20260925 14:55:10.929864 10968 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:10.929903 10968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:10.929986 10968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:10.930959 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "d5409261501649d998bfc81fbef36fae"
format_stamp: "Formatted at 2026-09-25 14:55:10 on dist-test-slave-1sv7"
server_key: "48549598d3eea95aa6957ddf08d9f4c2"
server_key_iv: "2d4d876b04357701cb01103441a1266d"
server_key_version: "encryptionkey@0"
I20260925 14:55:10.931280 10968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "d5409261501649d998bfc81fbef36fae"
format_stamp: "Formatted at 2026-09-25 14:55:10 on dist-test-slave-1sv7"
server_key: "48549598d3eea95aa6957ddf08d9f4c2"
server_key_iv: "2d4d876b04357701cb01103441a1266d"
server_key_version: "encryptionkey@0"
I20260925 14:55:10.932578 10968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:55:10.933241 10983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:10.933429 10968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:10.933516 10968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "d5409261501649d998bfc81fbef36fae"
format_stamp: "Formatted at 2026-09-25 14:55:10 on dist-test-slave-1sv7"
server_key: "48549598d3eea95aa6957ddf08d9f4c2"
server_key_iv: "2d4d876b04357701cb01103441a1266d"
server_key_version: "encryptionkey@0"
I20260925 14:55:10.933586 10968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:10.958485 10968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:10.961045 10968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:10.961206 10968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:10.961757 10968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:10.961793 10968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:10.961849 10968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:10.961879 10968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:10.968576 10968 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:40409
I20260925 14:55:10.968673 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:40409 every 8 connection(s)
I20260925 14:55:10.969039 10968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:55:10 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348110, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:10.972131 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.962299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:34361 (local address 127.9.141.126:40711)
0925 14:55:10.962349 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:10.962351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:10.962591 (+ 240us) server_negotiation.cc:407] Connection header received
0925 14:55:10.963173 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:10.963174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:10.963190 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:10.963252 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:10.964155 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.965013 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.966732 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.966928 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.966984 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.969691 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:10.969692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:10.969692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:10.969698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:10.971004 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.971372 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.971373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.971373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.971424 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.971587 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.971587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.971588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.971679 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:10.971723 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:10.971951 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:55:10.972020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:10.972190 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.962293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.9.141.65:34361)
0925 14:55:10.962465 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.962499 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:10.963002 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:10.963335 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:10.963340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:10.963714 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:10.964009 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:10.964016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.965124 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.965125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:10.966627 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:10.966631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.967018 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.967019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:10.967181 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.967781 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:10.967791 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:10.969582 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:10.971074 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:10.971079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:10.971081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:10.971304 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:10.971466 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:10.971467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:10.971467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:10.971550 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:10.971728 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:10.971731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:10.971841 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 14:55:10.971922 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:55:10.972036 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260925 14:55:10.972741 11099 heartbeater.cc:347] Connected to a master server at 127.9.141.126:40711
I20260925 14:55:10.972824 11099 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:10.973007 11099 heartbeater.cc:511] Master 127.9.141.126:40711 requested a full tablet report, sending...
I20260925 14:55:10.973541 10896 ts_manager.cc:194] Registered new tserver with Master: d5409261501649d998bfc81fbef36fae (127.9.141.65:40409)
I20260925 14:55:10.974243 10896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:34361
I20260925 14:55:10.978449 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 10968
I20260925 14:55:10.978524 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:10.978734 9781 external_mini_cluster.cc:1428] Setting key 627ebfb2f9c483708cbf57f522f3dee8
I20260925 14:55:10.979211 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.973731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.9.141.65:35863)
0925 14:55:10.973861 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:10.973871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:10.973954 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:10.974082 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:10.974083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:10.974261 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:10.974335 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:10.974337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.975009 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.975010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:10.975522 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:10.975523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.975920 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.975920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:10.975992 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.976426 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:10.976432 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:10.976732 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:10.978149 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:10.978149 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:10.978150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:10.978348 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:10.978585 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:10.978586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:10.978586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:10.978647 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:10.978920 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:10.978921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:10.978967 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:55:10.979037 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:55:10.979137 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260925 14:55:10.979247 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:10.973774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:35863 (local address 127.9.141.126:40711)
0925 14:55:10.973813 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:10.973816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:10.973939 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:55:10.974012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:10.974012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:10.974022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:10.974083 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:10.974405 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.974906 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.975624 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:10.975836 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:10.975919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:10.976843 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:10.976844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:10.976844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:10.976851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:10.978042 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.978427 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.978429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.978429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.978490 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:10.978717 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:10.978718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:10.978719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:10.978860 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:10.978921 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:10.979084 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:55:10.979165 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:11.004149 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40711
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:11.084868 11108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:11.085059 11108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:11.085088 11108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:11.086148 11108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:11.086179 11108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:11.086198 11108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:11.086215 11108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:11.087512 11108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:40711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:11.087682 11108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:11.087845 11108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:11.092810 11114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:11.092887 11108 server_base.cc:1034] running on GCE node
W20260925 14:55:11.092834 11113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:11.092816 11116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:11.093252 11108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:11.093444 11108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:11.094584 11108 hybrid_clock.cc:648] HybridClock initialized: now 1790348111094565 us; error 41 us; skew 500 ppm
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:11.097101 11108 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:11.097481 11108 webserver.cc:460] Webserver started at http://127.9.141.66:43125/ using document root <none> and password file <none>
I20260925 14:55:11.097656 11108 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:11.097694 11108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:11.097780 11108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:11.098774 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "db1a32dd268c4a70bebe6023be8c605d"
format_stamp: "Formatted at 2026-09-25 14:55:11 on dist-test-slave-1sv7"
server_key: "954e99ecfe453179a4a1aa8fc2d3164e"
server_key_iv: "0c1173e322030875f282ba6f435f5375"
server_key_version: "encryptionkey@0"
I20260925 14:55:11.099090 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "db1a32dd268c4a70bebe6023be8c605d"
format_stamp: "Formatted at 2026-09-25 14:55:11 on dist-test-slave-1sv7"
server_key: "954e99ecfe453179a4a1aa8fc2d3164e"
server_key_iv: "0c1173e322030875f282ba6f435f5375"
server_key_version: "encryptionkey@0"
I20260925 14:55:11.100345 11108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:55:11.101125 11123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.101275 11108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:11.101353 11108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "db1a32dd268c4a70bebe6023be8c605d"
format_stamp: "Formatted at 2026-09-25 14:55:11 on dist-test-slave-1sv7"
server_key: "954e99ecfe453179a4a1aa8fc2d3164e"
server_key_iv: "0c1173e322030875f282ba6f435f5375"
server_key_version: "encryptionkey@0"
I20260925 14:55:11.101398 11108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:11.113082 11108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:11.115285 11108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:11.115394 11108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:11.115845 11108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:11.115880 11108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.115902 11108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:11.115916 11108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.122630 11108 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:40043
I20260925 14:55:11.122735 11238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:40043 every 8 connection(s)
I20260925 14:55:11.123037 11108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:11.126801 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.116493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:56081 (local address 127.9.141.126:40711)
0925 14:55:11.116573 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.116575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.116798 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:55:11.117369 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.117370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.117382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.117462 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.118386 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.119213 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.120945 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.121186 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.121301 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.123922 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.123923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.123923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.123930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.125363 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.125777 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.125778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.125779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.125841 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.126139 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.126141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.126141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.126273 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.126347 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.126649 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.126734 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:55:11.126947 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.116499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.9.141.66:56081)
0925 14:55:11.116668 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.116708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.117248 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.117586 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.117592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.117993 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:11.118290 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.118297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.119316 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.119317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.120823 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.120826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.121319 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.121319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.121479 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.122076 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:11.122086 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:11.123835 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:11.125444 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.125450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.125452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.125682 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.125934 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.125935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.125935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.126041 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.126367 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:11.126372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:11.126496 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.126590 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.126716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260925 14:55:11.127578 11239 heartbeater.cc:347] Connected to a master server at 127.9.141.126:40711
I20260925 14:55:11.127709 11239 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:11.127921 11239 heartbeater.cc:511] Master 127.9.141.126:40711 requested a full tablet report, sending...
I20260925 14:55:11.128403 10896 ts_manager.cc:194] Registered new tserver with Master: db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043)
I20260925 14:55:11.128855 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 11108
I20260925 14:55:11.128954 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:11.128973 10896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:56081
I20260925 14:55:11.129161 9781 external_mini_cluster.cc:1428] Setting key bf64b3c6d46f1b538e8b80a5e8f93c64
I20260925 14:55:11.135026 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.128699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:47049 (local address 127.9.141.126:40711)
0925 14:55:11.128813 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.128815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.128928 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:55:11.129013 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.129014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.129026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.129092 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.129563 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.130168 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.130896 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.131175 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.131230 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.132291 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.132292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.132293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.132298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.133751 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.134214 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.134215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.134216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.134271 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.134508 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.134508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.134509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.134641 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.134697 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.134886 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.134969 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 14:55:11.135193 11245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.128632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.9.141.66:47049)
0925 14:55:11.128870 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.128880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.128952 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.129155 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.129157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.129390 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:11.129465 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.129467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.130264 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.130265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.130801 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.130802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.131250 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.131251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.131349 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.131775 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:11.131781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:11.132208 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:11.133841 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.133842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.133842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.134109 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.134355 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.134356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.134356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.134434 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.134716 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:11.134717 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:11.134775 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.134872 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.135018 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:11.156201 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40711
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:11.236737 11248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:11.236989 11248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:11.237032 11248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:11.238294 11248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:11.238365 11248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:11.238400 11248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:11.238438 11248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:11.239944 11248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:40711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:11.240240 11248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:11.240559 11248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:11.245978 11254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:11.246017 11253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:11.246006 11256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:11.246230 11248 server_base.cc:1034] running on GCE node
I20260925 14:55:11.246455 11248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:11.246680 11248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:11.247924 11248 hybrid_clock.cc:648] HybridClock initialized: now 1790348111247908 us; error 29 us; skew 500 ppm
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:11.250898 11248 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:11.251379 11248 webserver.cc:460] Webserver started at http://127.9.141.67:35153/ using document root <none> and password file <none>
I20260925 14:55:11.251600 11248 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:11.251645 11248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:11.251768 11248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:11.252877 11248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "a1130048adf04d3086521a1749776820"
format_stamp: "Formatted at 2026-09-25 14:55:11 on dist-test-slave-1sv7"
server_key: "0fd87c500bb00bef48fb2736767cdc80"
server_key_iv: "40da399f1ad05f9f72ee3882ac653964"
server_key_version: "encryptionkey@0"
I20260925 14:55:11.253245 11248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "a1130048adf04d3086521a1749776820"
format_stamp: "Formatted at 2026-09-25 14:55:11 on dist-test-slave-1sv7"
server_key: "0fd87c500bb00bef48fb2736767cdc80"
server_key_iv: "40da399f1ad05f9f72ee3882ac653964"
server_key_version: "encryptionkey@0"
I20260925 14:55:11.254628 11248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:11.255563 11263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.255764 11248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:11.255836 11248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "a1130048adf04d3086521a1749776820"
format_stamp: "Formatted at 2026-09-25 14:55:11 on dist-test-slave-1sv7"
server_key: "0fd87c500bb00bef48fb2736767cdc80"
server_key_iv: "40da399f1ad05f9f72ee3882ac653964"
server_key_version: "encryptionkey@0"
I20260925 14:55:11.255925 11248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:11.273700 11248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:11.275755 11248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:11.275897 11248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:11.276446 11248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:11.276490 11248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.276513 11248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:11.276525 11248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.283089 11248 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:36451
I20260925 14:55:11.283161 11378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:36451 every 8 connection(s)
I20260925 14:55:11.283452 11248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:11.287031 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.276919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:59059 (local address 127.9.141.126:40711)
0925 14:55:11.277020 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.277022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.277209 (+ 187us) server_negotiation.cc:407] Connection header received
0925 14:55:11.277764 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.277766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.277781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.277861 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.278766 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.279601 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.281209 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.281530 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.281616 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.284210 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.284212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.284213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.284220 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.285651 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.286074 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.286075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.286076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.286130 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.286388 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.286389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.286390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.286511 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.286583 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.286855 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.286932 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:55:11.287206 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.276922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.9.141.67:59059)
0925 14:55:11.277092 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.277126 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.277625 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.277944 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.277948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.278365 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:11.278658 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.278663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.279703 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.279704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.281119 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.281122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.281625 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.281626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.281792 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.282382 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:11.282395 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:11.284110 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:11.285757 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.285763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.285765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.285983 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.286207 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.286208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.286208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.286319 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.286584 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:11.286591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:11.286725 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.286832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.286985 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260925 14:55:11.287683 11379 heartbeater.cc:347] Connected to a master server at 127.9.141.126:40711
I20260925 14:55:11.287766 11379 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:11.287962 11379 heartbeater.cc:511] Master 127.9.141.126:40711 requested a full tablet report, sending...
I20260925 14:55:11.288395 10894 ts_manager.cc:194] Registered new tserver with Master: a1130048adf04d3086521a1749776820 (127.9.141.67:36451)
I20260925 14:55:11.288964 10894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:59059
I20260925 14:55:11.291003 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 11248
I20260925 14:55:11.291108 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:11.291285 9781 external_mini_cluster.cc:1428] Setting key 25f2567a219a21c562d10d1c5c56f6aa
I20260925 14:55:11.292208 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:11.294605 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.288726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.9.141.67:35071)
0925 14:55:11.288849 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.288859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.288908 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.289060 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.289061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.289312 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:11.289370 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.289372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.290041 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.290042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.290522 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.290523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.290859 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.290860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.290913 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.291367 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:11.291372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:11.291716 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:11.293392 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.293393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.293393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.293634 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.293928 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.293929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.293929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.293992 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.294327 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:11.294328 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:11.294370 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.294447 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.294539 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260925 14:55:11.294687 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.288797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:35071 (local address 127.9.141.126:40711)
0925 14:55:11.288834 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.288836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.288896 (+ 60us) server_negotiation.cc:407] Connection header received
0925 14:55:11.288971 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.288972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.288980 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.289052 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.289496 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.289933 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.290595 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.290768 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.290855 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.291775 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.291776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.291777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.291784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.293307 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.293762 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.293764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.293764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.293838 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.294061 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.294062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.294062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.294225 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.294299 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.294495 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.294580 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:11.316214 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.308482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.9.141.126:40711)
0925 14:55:11.308547 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.308549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.308566 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:55:11.308590 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.308590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.308599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.308664 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.309288 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.309763 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.310814 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.311055 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.311118 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.313645 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.313647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.313647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.313653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.315132 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.315486 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.315487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.315488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.315543 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.315726 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.315727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.315727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.315837 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.315927 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.316082 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.316149 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:55:11.317983 10894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:11.319605 10894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:11.326709 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.322318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36451 (local address 127.0.0.1:54160)
0925 14:55:11.322934 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.322944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.322987 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.323284 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.323284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.323286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.323334 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.323335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.324237 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.324238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.325607 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.325610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.326361 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.326362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.326440 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.326455 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.326557 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.326612 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":77,"thread_start_us":135,"threads_started":1}
I20260925 14:55:11.326834 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.322391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.9.141.66:40043)
0925 14:55:11.322585 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.322589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.322601 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:11.322753 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.322755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.322770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.322848 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.323064 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.324256 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.325791 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.326474 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.326538 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.326583 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.326688 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.326732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260925 14:55:11.326833 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.322318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:40409 (local address 127.0.0.1:43530)
0925 14:55:11.322638 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.322648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.322688 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.323341 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.323342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.323344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.323408 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.323411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.324662 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.324663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.325559 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.325562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.326603 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.326604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.326668 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.326673 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.326722 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.326761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":160,"threads_started":1}
I20260925 14:55:11.326998 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.322313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:40043 (local address 127.0.0.1:53394)
0925 14:55:11.322508 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.322552 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.322625 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.322875 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.322879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.322889 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.322965 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.322967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.324330 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.324331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.325719 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.325721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.326555 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.326556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.326620 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.326624 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.326678 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.326710 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":6,"thread_start_us":49,"threads_started":1}
I20260925 14:55:11.327080 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.322610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.9.141.65:40409)
0925 14:55:11.322775 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.322779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.322790 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:11.322834 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.322836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.322870 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.322956 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.323486 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.324581 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.325650 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.326427 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.326504 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.326565 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.326959 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.326997 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260925 14:55:11.327296 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.322600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.9.141.67:36451)
0925 14:55:11.323079 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.323084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.323094 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:11.323170 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.323172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.323187 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.323266 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.323418 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.324141 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.325702 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.326255 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.326357 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.326420 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.326754 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.327205 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":88,"threads_started":1}
I20260925 14:55:11.327780 11033 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1842d3e6d243f5accdcfca4ec93478 (DEFAULT_TABLE table=test-table [id=ac1554b227e44418bef2b95abeccceaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:11.327898 11173 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1842d3e6d243f5accdcfca4ec93478 (DEFAULT_TABLE table=test-table [id=ac1554b227e44418bef2b95abeccceaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:11.327934 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1842d3e6d243f5accdcfca4ec93478 (DEFAULT_TABLE table=test-table [id=ac1554b227e44418bef2b95abeccceaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:11.328161 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1842d3e6d243f5accdcfca4ec93478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.328235 11173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1842d3e6d243f5accdcfca4ec93478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.328238 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1842d3e6d243f5accdcfca4ec93478. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.331108 11402 tablet_bootstrap.cc:492] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Bootstrap starting.
I20260925 14:55:11.331739 11404 tablet_bootstrap.cc:492] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Bootstrap starting.
I20260925 14:55:11.331879 11402 tablet_bootstrap.cc:654] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.332172 11403 tablet_bootstrap.cc:492] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Bootstrap starting.
I20260925 14:55:11.332199 11402 log.cc:824] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:11.332399 11404 tablet_bootstrap.cc:654] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.332794 11404 log.cc:824] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:11.332973 11402 tablet_bootstrap.cc:492] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: No bootstrap required, opened a new log
I20260925 14:55:11.333036 11402 ts_tablet_manager.cc:1397] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 14:55:11.333129 11403 tablet_bootstrap.cc:654] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.333449 11403 log.cc:824] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:11.333583 11404 tablet_bootstrap.cc:492] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: No bootstrap required, opened a new log
I20260925 14:55:11.333657 11404 ts_tablet_manager.cc:1397] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:11.334589 11402 raft_consensus.cc:348] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.334748 11402 raft_consensus.cc:374] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.334793 11402 raft_consensus.cc:729] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5409261501649d998bfc81fbef36fae, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.334929 11402 consensus_queue.cc:260] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.335201 11402 ts_tablet_manager.cc:1428] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:55:11.335191 11404 raft_consensus.cc:348] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.335356 11404 raft_consensus.cc:374] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.335357 11099 heartbeater.cc:503] Master 127.9.141.126:40711 was elected leader, sending a full tablet report...
I20260925 14:55:11.335404 11404 raft_consensus.cc:729] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a32dd268c4a70bebe6023be8c605d, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.335510 11404 consensus_queue.cc:260] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.335768 11404 ts_tablet_manager.cc:1428] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:11.335781 11239 heartbeater.cc:503] Master 127.9.141.126:40711 was elected leader, sending a full tablet report...
I20260925 14:55:11.336467 11403 tablet_bootstrap.cc:492] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: No bootstrap required, opened a new log
I20260925 14:55:11.336539 11403 ts_tablet_manager.cc:1397] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:55:11.338122 11403 raft_consensus.cc:348] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.338265 11403 raft_consensus.cc:374] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.338310 11403 raft_consensus.cc:729] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1130048adf04d3086521a1749776820, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.338421 11403 consensus_queue.cc:260] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.338661 11403 ts_tablet_manager.cc:1428] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:11.338732 11379 heartbeater.cc:503] Master 127.9.141.126:40711 was elected leader, sending a full tablet report...
I20260925 14:55:11.341475 11408 raft_consensus.cc:482] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:11.341554 11408 raft_consensus.cc:504] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.341873 11408 leader_election.cc:288] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043), a1130048adf04d3086521a1749776820 (127.9.141.67:36451)
I20260925 14:55:11.344835 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.342147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:42187 (local address 127.9.141.67:36451)
0925 14:55:11.342188 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.342190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.342320 (+ 130us) server_negotiation.cc:407] Connection header received
0925 14:55:11.342388 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.342389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.342398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.342463 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.342610 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.343084 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.344072 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.344505 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.344574 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.344583 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.344738 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.344770 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:55:11.344892 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.342100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36451 (local address 127.9.141.65:42187)
0925 14:55:11.342243 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.342253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.342337 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.342467 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.342468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.342469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.342534 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.342535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.343167 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.343168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.343976 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.343979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.344580 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.344581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.344669 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.344673 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.344731 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.344770 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260925 14:55:11.344908 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.342100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:40043 (local address 127.9.141.65:43101)
0925 14:55:11.342155 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.342168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.342208 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.342485 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.342486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.342488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.342547 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.342549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.343172 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.343173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.343971 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.343973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.344629 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.344630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.344721 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.344724 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.344773 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.344822 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":23}
I20260925 14:55:11.345118 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.342289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:43101 (local address 127.9.141.66:40043)
0925 14:55:11.342345 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.342347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.342361 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:55:11.342388 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.342392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.342401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.342467 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.342622 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.343088 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.344073 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.344520 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.344588 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.344597 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.345013 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.345061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:11.345208 11333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1842d3e6d243f5accdcfca4ec93478" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1130048adf04d3086521a1749776820" is_pre_election: true
I20260925 14:55:11.345352 11333 raft_consensus.cc:2454] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 0.
I20260925 14:55:11.345482 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1842d3e6d243f5accdcfca4ec93478" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a32dd268c4a70bebe6023be8c605d" is_pre_election: true
I20260925 14:55:11.345561 10986 leader_election.cc:302] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1130048adf04d3086521a1749776820, d5409261501649d998bfc81fbef36fae; no voters:
I20260925 14:55:11.345608 11193 raft_consensus.cc:2454] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 0.
I20260925 14:55:11.345686 11408 raft_consensus.cc:2781] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:11.345763 11408 raft_consensus.cc:482] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:11.345781 11408 raft_consensus.cc:3037] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.346442 11408 raft_consensus.cc:504] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.346575 11408 leader_election.cc:288] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 election: Requested vote from peers db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043), a1130048adf04d3086521a1749776820 (127.9.141.67:36451)
I20260925 14:55:11.346719 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1842d3e6d243f5accdcfca4ec93478" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a32dd268c4a70bebe6023be8c605d"
I20260925 14:55:11.346797 11193 raft_consensus.cc:3037] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.347417 11193 raft_consensus.cc:2454] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 1.
I20260925 14:55:11.347546 11333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1842d3e6d243f5accdcfca4ec93478" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1130048adf04d3086521a1749776820"
I20260925 14:55:11.347625 11333 raft_consensus.cc:3037] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.348007 10985 leader_election.cc:302] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5409261501649d998bfc81fbef36fae, db1a32dd268c4a70bebe6023be8c605d; no voters:
I20260925 14:55:11.348135 11408 raft_consensus.cc:2781] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:11.348286 11408 raft_consensus.cc:686] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 1 LEADER]: Becoming Leader. State: Replica: d5409261501649d998bfc81fbef36fae, State: Running, Role: LEADER
I20260925 14:55:11.348403 11408 consensus_queue.cc:237] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.348603 11333 raft_consensus.cc:2454] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 1.
I20260925 14:55:11.349231 10894 catalog_manager.cc:5495] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae reported cstate change: term changed from 0 to 1, leader changed from <none> to d5409261501649d998bfc81fbef36fae (127.9.141.65). New cstate: current_term: 1 leader_uuid: "d5409261501649d998bfc81fbef36fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:11.356426 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.353893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.9.141.65:40409)
0925 14:55:11.353989 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.353991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.354001 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:11.354067 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.354068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.354083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.354140 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:11.354385 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.354843 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.355666 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.355840 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.355894 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.356070 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.356152 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.356212 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.356297 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.356314 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:55:11.358098 11193 raft_consensus.cc:1261] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Refusing update from remote peer d5409261501649d998bfc81fbef36fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:11.358163 11333 raft_consensus.cc:1261] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Refusing update from remote peer d5409261501649d998bfc81fbef36fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:11.358439 11412 consensus_queue.cc:1035] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:11.358534 11408 consensus_queue.cc:1035] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:11.360996 11416 mvcc.cc:205] Tried to move back new op lower bound from 7333265864120209408 to 7333265864084013056. Current Snapshot: MvccSnapshot[applied={T|T < 7333265864120209408}]
I20260925 14:55:11.361397 11417 mvcc.cc:205] Tried to move back new op lower bound from 7333265864120209408 to 7333265864084013056. Current Snapshot: MvccSnapshot[applied={T|T < 7333265864120209408}]
I20260925 14:55:11.364584 10896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:11.364682 10896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:11.365433 10896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: Sending DeleteTablet for 3 replicas of tablet 1a1842d3e6d243f5accdcfca4ec93478
I20260925 14:55:11.365787 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1842d3e6d243f5accdcfca4ec93478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:11 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:43530
I20260925 14:55:11.365844 11173 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1842d3e6d243f5accdcfca4ec93478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:11 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260925 14:55:11.365835 11313 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1842d3e6d243f5accdcfca4ec93478 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:11 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:54160
I20260925 14:55:11.365988 11425 tablet_replica.cc:330] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: stopping tablet replica
I20260925 14:55:11.366072 11425 raft_consensus.cc:2229] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:11.366191 11425 raft_consensus.cc:2258] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:11.366284 11426 tablet_replica.cc:330] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: stopping tablet replica
I20260925 14:55:11.366308 10896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 14:55:11.366384 11426 raft_consensus.cc:2229] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 14:55:11.366454 10896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:11.366487 11426 raft_consensus.cc:2258] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:11.366636 11427 tablet_replica.cc:330] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: stopping tablet replica
I20260925 14:55:11.366731 11427 raft_consensus.cc:2229] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:11.366834 11427 raft_consensus.cc:2258] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:11.367332 11427 ts_tablet_manager.cc:1905] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:11.367646 11033 tablet_service.cc:1467] Processing CreateTablet for tablet 65226da7051a4c77ac5f2676d19584fa (DEFAULT_TABLE table=test-table [id=57633aa5b05c4195aa2db1a86aca8d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:11.367767 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65226da7051a4c77ac5f2676d19584fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.369124 11426 ts_tablet_manager.cc:1905] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:11.369263 11402 tablet_bootstrap.cc:492] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Bootstrap starting.
I20260925 14:55:11.369665 11402 tablet_bootstrap.cc:654] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.370121 11173 tablet_service.cc:1467] Processing CreateTablet for tablet 65226da7051a4c77ac5f2676d19584fa (DEFAULT_TABLE table=test-table [id=57633aa5b05c4195aa2db1a86aca8d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:11.370234 11173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65226da7051a4c77ac5f2676d19584fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.370412 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 65226da7051a4c77ac5f2676d19584fa (DEFAULT_TABLE table=test-table [id=57633aa5b05c4195aa2db1a86aca8d45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:11.370530 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65226da7051a4c77ac5f2676d19584fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.371214 11404 tablet_bootstrap.cc:492] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Bootstrap starting.
I20260925 14:55:11.371601 11404 tablet_bootstrap.cc:654] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.371894 11403 tablet_bootstrap.cc:492] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Bootstrap starting.
I20260925 14:55:11.371981 11402 tablet_bootstrap.cc:492] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: No bootstrap required, opened a new log
I20260925 14:55:11.372023 11402 ts_tablet_manager.cc:1397] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 14:55:11.372187 11402 raft_consensus.cc:348] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.372259 11402 raft_consensus.cc:374] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.372280 11402 raft_consensus.cc:729] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5409261501649d998bfc81fbef36fae, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.372310 11403 tablet_bootstrap.cc:654] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.372319 11402 consensus_queue.cc:260] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.372421 11402 ts_tablet_manager.cc:1428] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:11.372859 11425 ts_tablet_manager.cc:1905] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:11.374217 11426 ts_tablet_manager.cc:1918] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:11.374217 11425 ts_tablet_manager.cc:1918] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:11.374301 11425 log.cc:1196] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/1a1842d3e6d243f5accdcfca4ec93478
I20260925 14:55:11.374305 11426 log.cc:1196] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/1a1842d3e6d243f5accdcfca4ec93478
I20260925 14:55:11.374548 11426 ts_tablet_manager.cc:1939] T 1a1842d3e6d243f5accdcfca4ec93478 P db1a32dd268c4a70bebe6023be8c605d: Deleting consensus metadata
I20260925 14:55:11.374583 11425 ts_tablet_manager.cc:1939] T 1a1842d3e6d243f5accdcfca4ec93478 P d5409261501649d998bfc81fbef36fae: Deleting consensus metadata
I20260925 14:55:11.374961 10883 catalog_manager.cc:4841] TS d5409261501649d998bfc81fbef36fae (127.9.141.65:40409): tablet 1a1842d3e6d243f5accdcfca4ec93478 (table test-table [id=ac1554b227e44418bef2b95abeccceaf]) successfully deleted
I20260925 14:55:11.374958 10881 catalog_manager.cc:4841] TS db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043): tablet 1a1842d3e6d243f5accdcfca4ec93478 (table test-table [id=ac1554b227e44418bef2b95abeccceaf]) successfully deleted
I20260925 14:55:11.375164 11427 ts_tablet_manager.cc:1918] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:11.375228 11427 log.cc:1196] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/1a1842d3e6d243f5accdcfca4ec93478
I20260925 14:55:11.375473 11427 ts_tablet_manager.cc:1939] T 1a1842d3e6d243f5accdcfca4ec93478 P a1130048adf04d3086521a1749776820: Deleting consensus metadata
I20260925 14:55:11.375912 11403 tablet_bootstrap.cc:492] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: No bootstrap required, opened a new log
I20260925 14:55:11.375960 11403 ts_tablet_manager.cc:1397] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 14:55:11.376060 11404 tablet_bootstrap.cc:492] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: No bootstrap required, opened a new log
I20260925 14:55:11.376093 11404 ts_tablet_manager.cc:1397] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:55:11.376113 11403 raft_consensus.cc:348] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.376137 10882 catalog_manager.cc:4841] TS a1130048adf04d3086521a1749776820 (127.9.141.67:36451): tablet 1a1842d3e6d243f5accdcfca4ec93478 (table test-table [id=ac1554b227e44418bef2b95abeccceaf]) successfully deleted
I20260925 14:55:11.376185 11403 raft_consensus.cc:374] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.376204 11403 raft_consensus.cc:729] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1130048adf04d3086521a1749776820, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.376220 11404 raft_consensus.cc:348] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.376246 11403 consensus_queue.cc:260] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.376281 11404 raft_consensus.cc:374] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.376303 11404 raft_consensus.cc:729] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a32dd268c4a70bebe6023be8c605d, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.376327 11403 ts_tablet_manager.cc:1428] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.376350 11404 consensus_queue.cc:260] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.376464 11404 ts_tablet_manager.cc:1428] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.429165 11420 raft_consensus.cc:482] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:11.429270 11420 raft_consensus.cc:504] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.429430 11420 leader_election.cc:288] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043), a1130048adf04d3086521a1749776820 (127.9.141.67:36451)
I20260925 14:55:11.429632 11333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65226da7051a4c77ac5f2676d19584fa" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1130048adf04d3086521a1749776820" is_pre_election: true
I20260925 14:55:11.429636 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65226da7051a4c77ac5f2676d19584fa" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a32dd268c4a70bebe6023be8c605d" is_pre_election: true
I20260925 14:55:11.429731 11333 raft_consensus.cc:2454] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 0.
I20260925 14:55:11.429747 11193 raft_consensus.cc:2454] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 0.
I20260925 14:55:11.429873 10986 leader_election.cc:302] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1130048adf04d3086521a1749776820, d5409261501649d998bfc81fbef36fae; no voters:
I20260925 14:55:11.429977 11420 raft_consensus.cc:2781] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:11.430009 11420 raft_consensus.cc:482] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:11.430037 11420 raft_consensus.cc:3037] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.430706 11420 raft_consensus.cc:504] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.430851 11420 leader_election.cc:288] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 election: Requested vote from peers db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043), a1130048adf04d3086521a1749776820 (127.9.141.67:36451)
I20260925 14:55:11.431033 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65226da7051a4c77ac5f2676d19584fa" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a32dd268c4a70bebe6023be8c605d"
I20260925 14:55:11.431031 11333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65226da7051a4c77ac5f2676d19584fa" candidate_uuid: "d5409261501649d998bfc81fbef36fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1130048adf04d3086521a1749776820"
I20260925 14:55:11.431109 11333 raft_consensus.cc:3037] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.431109 11193 raft_consensus.cc:3037] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.431686 11193 raft_consensus.cc:2454] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 1.
I20260925 14:55:11.431869 10985 leader_election.cc:302] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5409261501649d998bfc81fbef36fae, db1a32dd268c4a70bebe6023be8c605d; no voters:
I20260925 14:55:11.431936 11333 raft_consensus.cc:2454] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5409261501649d998bfc81fbef36fae in term 1.
I20260925 14:55:11.431982 11420 raft_consensus.cc:2781] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:11.432027 11420 raft_consensus.cc:686] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 1 LEADER]: Becoming Leader. State: Replica: d5409261501649d998bfc81fbef36fae, State: Running, Role: LEADER
I20260925 14:55:11.432080 11420 consensus_queue.cc:237] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.432739 10894 catalog_manager.cc:5495] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae reported cstate change: term changed from 0 to 1, leader changed from <none> to d5409261501649d998bfc81fbef36fae (127.9.141.65). New cstate: current_term: 1 leader_uuid: "d5409261501649d998bfc81fbef36fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348110, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:11.469669 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.461288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.9.141.126:40711)
0925 14:55:11.461345 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.461347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.461478 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:55:11.461557 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.461558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.461568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.461627 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.462252 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.462805 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.463817 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.464057 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.464130 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.466438 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.466440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.466440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.466447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.468073 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.468624 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.468625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.468625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.468687 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.469041 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.469042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.469042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.469182 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.469265 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.469426 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.469511 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:55:11.469734 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.461193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40711 (local address 127.0.0.1:53144)
0925 14:55:11.461389 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.461402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.461491 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.461657 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.461658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.462065 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:11.462124 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.462127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.462901 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.462902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.463706 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.463709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.464150 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.464151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.464217 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.464798 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:11.464803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:11.466334 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:11.468240 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.468247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.468249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.468528 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.468789 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:11.468790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:11.468790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:11.468944 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:11.469263 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:11.469271 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:11.469317 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.469404 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.469566 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":67,"thread_start_us":104,"threads_started":1}
W20260925 14:55:11.469740 11100 tablet.cc:2369] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:11.471782 10894 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:53144:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:55:11.471982 10894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:11.473398 11173 tablet_service.cc:1467] Processing CreateTablet for tablet 3d895a8b10b04754af993247ea79ecb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9de4c4383674ad98ee9989028adaeb5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:11.473397 11033 tablet_service.cc:1467] Processing CreateTablet for tablet 3d895a8b10b04754af993247ea79ecb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9de4c4383674ad98ee9989028adaeb5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:11.473461 11313 tablet_service.cc:1467] Processing CreateTablet for tablet 3d895a8b10b04754af993247ea79ecb8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9de4c4383674ad98ee9989028adaeb5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:11.473544 11173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d895a8b10b04754af993247ea79ecb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.473549 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d895a8b10b04754af993247ea79ecb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.473619 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d895a8b10b04754af993247ea79ecb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:11.474995 11402 tablet_bootstrap.cc:492] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: Bootstrap starting.
I20260925 14:55:11.475005 11404 tablet_bootstrap.cc:492] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: Bootstrap starting.
I20260925 14:55:11.475131 11403 tablet_bootstrap.cc:492] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: Bootstrap starting.
I20260925 14:55:11.475314 11402 tablet_bootstrap.cc:654] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.475414 11404 tablet_bootstrap.cc:654] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.475581 11403 tablet_bootstrap.cc:654] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:11.476070 11404 tablet_bootstrap.cc:492] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: No bootstrap required, opened a new log
I20260925 14:55:11.476115 11404 ts_tablet_manager.cc:1397] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:11.476254 11404 raft_consensus.cc:348] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.476311 11404 raft_consensus.cc:374] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.476325 11404 raft_consensus.cc:729] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1a32dd268c4a70bebe6023be8c605d, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.476384 11404 consensus_queue.cc:260] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.476435 11403 tablet_bootstrap.cc:492] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: No bootstrap required, opened a new log
I20260925 14:55:11.476490 11404 ts_tablet_manager.cc:1428] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.476475 11403 ts_tablet_manager.cc:1397] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:11.476613 11403 raft_consensus.cc:348] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.476686 11403 raft_consensus.cc:374] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.476711 11403 raft_consensus.cc:729] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1130048adf04d3086521a1749776820, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.476763 11403 consensus_queue.cc:260] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.476843 11403 ts_tablet_manager.cc:1428] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.476976 11409 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.477088 11409 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.477257 11410 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.477349 11410 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.478222 11402 tablet_bootstrap.cc:492] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: No bootstrap required, opened a new log
I20260925 14:55:11.478266 11402 ts_tablet_manager.cc:1397] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 14:55:11.478340 11402 raft_consensus.cc:348] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.478389 11402 raft_consensus.cc:374] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:11.478401 11402 raft_consensus.cc:729] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5409261501649d998bfc81fbef36fae, State: Initialized, Role: FOLLOWER
I20260925 14:55:11.478461 11402 consensus_queue.cc:260] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.478552 11402 ts_tablet_manager.cc:1428] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:11.478564 11408 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.478631 11408 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.523494 11410 raft_consensus.cc:482] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:11.523582 11410 raft_consensus.cc:504] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.524019 11410 leader_election.cc:288] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5409261501649d998bfc81fbef36fae (127.9.141.65:40409), db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043)
I20260925 14:55:11.527128 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.524204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:40043 (local address 127.9.141.67:55875)
0925 14:55:11.524255 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.524266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.524307 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.524539 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.524540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.524542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.524612 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.524614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.525260 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.525262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.526000 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.526002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.526837 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.526838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.526926 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.526936 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.527010 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.527052 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:55:11.527128 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.524265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:55875 (local address 127.9.141.66:40043)
0925 14:55:11.524307 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.524309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.524319 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:11.524449 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.524450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.524466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.524534 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.524695 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.525176 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.526319 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.526741 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.526842 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.526852 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.527015 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.527053 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:55:11.527438 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d895a8b10b04754af993247ea79ecb8" candidate_uuid: "a1130048adf04d3086521a1749776820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a32dd268c4a70bebe6023be8c605d" is_pre_election: true
I20260925 14:55:11.527571 11193 raft_consensus.cc:2454] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1130048adf04d3086521a1749776820 in term 0.
I20260925 14:55:11.527778 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.524342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:36115 (local address 127.9.141.65:40409)
0925 14:55:11.524418 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.524420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.524567 (+ 147us) server_negotiation.cc:407] Connection header received
0925 14:55:11.524642 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.524643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.524653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.524728 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:11.524910 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.525420 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.526754 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.527383 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.527457 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.527465 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.527650 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.527710 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:11.527773 11265 leader_election.cc:302] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1130048adf04d3086521a1749776820, db1a32dd268c4a70bebe6023be8c605d; no voters:
I20260925 14:55:11.527905 11410 raft_consensus.cc:2781] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:11.528061 11410 raft_consensus.cc:482] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:11.528138 11410 raft_consensus.cc:3037] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.528241 11441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.524238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:40409 (local address 127.9.141.67:36115)
0925 14:55:11.524453 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.524470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.524582 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.524746 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.524747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.524749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:11.524828 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.524830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.525515 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.525516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.526652 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.526655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.527463 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.527464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.527540 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.527545 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.527869 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.527917 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":120,"threads_started":1}
I20260925 14:55:11.528568 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d895a8b10b04754af993247ea79ecb8" candidate_uuid: "a1130048adf04d3086521a1749776820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5409261501649d998bfc81fbef36fae" is_pre_election: true
I20260925 14:55:11.528757 11053 raft_consensus.cc:2454] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1130048adf04d3086521a1749776820 in term 0.
I20260925 14:55:11.529170 11410 raft_consensus.cc:504] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.529326 11410 leader_election.cc:288] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [CANDIDATE]: Term 1 election: Requested vote from peers d5409261501649d998bfc81fbef36fae (127.9.141.65:40409), db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043)
I20260925 14:55:11.529484 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d895a8b10b04754af993247ea79ecb8" candidate_uuid: "a1130048adf04d3086521a1749776820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5409261501649d998bfc81fbef36fae"
I20260925 14:55:11.529496 11193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d895a8b10b04754af993247ea79ecb8" candidate_uuid: "a1130048adf04d3086521a1749776820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1a32dd268c4a70bebe6023be8c605d"
I20260925 14:55:11.529569 11053 raft_consensus.cc:3037] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.529592 11193 raft_consensus.cc:3037] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:11.530169 11193 raft_consensus.cc:2454] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1130048adf04d3086521a1749776820 in term 1.
I20260925 14:55:11.530318 11053 raft_consensus.cc:2454] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1130048adf04d3086521a1749776820 in term 1.
I20260925 14:55:11.530422 11265 leader_election.cc:302] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1130048adf04d3086521a1749776820, db1a32dd268c4a70bebe6023be8c605d; no voters:
I20260925 14:55:11.530542 11410 raft_consensus.cc:2781] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:11.530719 11410 raft_consensus.cc:686] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [term 1 LEADER]: Becoming Leader. State: Replica: a1130048adf04d3086521a1749776820, State: Running, Role: LEADER
I20260925 14:55:11.530792 11410 consensus_queue.cc:237] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } }
I20260925 14:55:11.531221 11442 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: TxnStatusTablet state changed. Reason: New leader a1130048adf04d3086521a1749776820. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.531307 11442 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:11.531486 11444 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:55:11.531635 11444 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:55:11.531615 10894 catalog_manager.cc:5495] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1130048adf04d3086521a1749776820 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } health_report { overall_health: HEALTHY } } }
W20260925 14:55:11.534008 11380 tablet.cc:2369] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:11.549603 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.546190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.9.141.67:36451)
0925 14:55:11.546236 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.546238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.546364 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:55:11.546404 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.546405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.546426 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.546502 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:11.546645 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.547468 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.548736 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.549007 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.549069 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.549301 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.549389 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.549439 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.549530 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.549546 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:11.549677 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.546130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36451 (local address 127.0.0.1:54164)
0925 14:55:11.546214 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.546230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.546289 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.546506 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.546507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.546509 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:11.546566 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.546569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.547562 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.547563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.548633 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.548636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.549093 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.549095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.549170 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.549189 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.549458 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.549466 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.549528 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.549581 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:55:11.551167 11193 raft_consensus.cc:1261] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Refusing update from remote peer a1130048adf04d3086521a1749776820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:11.551328 11053 raft_consensus.cc:1261] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Refusing update from remote peer a1130048adf04d3086521a1749776820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:11.551643 11410 consensus_queue.cc:1035] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:11.551828 11410 consensus_queue.cc:1035] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:11.552366 11408 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: TxnStatusTablet state changed. Reason: New leader a1130048adf04d3086521a1749776820. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.552481 11408 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.552443 11409 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: TxnStatusTablet state changed. Reason: New leader a1130048adf04d3086521a1749776820. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.552532 11409 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.553110 11410 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.553195 11410 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:11.553287 11410 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.553301 11415 mvcc.cc:205] Tried to move back new op lower bound from 7333265864911380480 to 7333265864831004672. Current Snapshot: MvccSnapshot[applied={T|T < 7333265864911380480}]
I20260925 14:55:11.553337 11410 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P a1130048adf04d3086521a1749776820: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:11.553391 11408 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.553457 11408 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P d5409261501649d998bfc81fbef36fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.554010 11409 tablet_replica.cc:439] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1130048adf04d3086521a1749776820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5409261501649d998bfc81fbef36fae" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 40409 } } peers { permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 } } peers { permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 } } }
I20260925 14:55:11.554097 11409 tablet_replica.cc:442] T 3d895a8b10b04754af993247ea79ecb8 P db1a32dd268c4a70bebe6023be8c605d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:11.559715 11451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.556908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36451 (local address 127.9.141.65:50821)
0925 14:55:11.557052 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.557062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.557112 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.557239 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.557240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.557241 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:11.557280 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.557281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.558246 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.558247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.558868 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.558870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.559346 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.559347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.559387 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.559399 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.559592 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.559593 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.559626 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.559662 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260925 14:55:11.559724 11401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.557067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:50821 (local address 127.9.141.67:36451)
0925 14:55:11.557114 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.557116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.557150 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:55:11.557173 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.557173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.557183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.557230 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:11.557322 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.558167 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.558939 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.559267 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.559348 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.559477 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.559533 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.559593 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.559630 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.559650 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:55:11.561834 11193 raft_consensus.cc:1261] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Refusing update from remote peer d5409261501649d998bfc81fbef36fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:11.561857 11333 raft_consensus.cc:1261] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Refusing update from remote peer d5409261501649d998bfc81fbef36fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:11.562044 11420 consensus_queue.cc:1035] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1130048adf04d3086521a1749776820" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:11.562152 11408 consensus_queue.cc:1035] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1a32dd268c4a70bebe6023be8c605d" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 40043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:11.570839 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.567558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:40409 (local address 127.9.141.67:36131)
0925 14:55:11.567674 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:11.567685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:11.567721 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:11.567858 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:11.567858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:11.567859 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:11.567906 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.567907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.568889 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.568890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.569857 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:11.569860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.570432 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.570433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:11.570508 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.570529 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.570723 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.570724 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:55:11.570756 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 14:55:11.570789 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 14:55:11.570870 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.567646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:36131 (local address 127.9.141.65:40409)
0925 14:55:11.567712 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.567714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.567725 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:11.567776 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.567777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.567787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.567858 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:11.567945 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.568789 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.569941 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.570280 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.570371 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.570618 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:11.570655 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:11.570716 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.570772 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.570795 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:55:11.579555 10896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:11.579674 10896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:11.580420 10896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dce37ecfdd84491b64f098844a5e491: Sending DeleteTablet for 3 replicas of tablet 65226da7051a4c77ac5f2676d19584fa
I20260925 14:55:11.580664 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet 65226da7051a4c77ac5f2676d19584fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:11 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:43530
I20260925 14:55:11.580772 11425 tablet_replica.cc:330] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: stopping tablet replica
I20260925 14:55:11.580767 11313 tablet_service.cc:1514] Processing DeleteTablet for tablet 65226da7051a4c77ac5f2676d19584fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:11 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:54160
I20260925 14:55:11.580837 11425 raft_consensus.cc:2229] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:11.580883 11427 tablet_replica.cc:330] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: stopping tablet replica
I20260925 14:55:11.580891 11425 raft_consensus.cc:2258] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:11.580936 11427 raft_consensus.cc:2229] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:11.580929 11173 tablet_service.cc:1514] Processing DeleteTablet for tablet 65226da7051a4c77ac5f2676d19584fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:11 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260925 14:55:11.580972 11427 raft_consensus.cc:2258] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:11.581013 11426 tablet_replica.cc:330] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: stopping tablet replica
I20260925 14:55:11.581053 11426 raft_consensus.cc:2229] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:11.581086 11426 raft_consensus.cc:2258] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:11.581374 11425 ts_tablet_manager.cc:1905] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:11.581418 11426 ts_tablet_manager.cc:1905] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:11.582895 11427 ts_tablet_manager.cc:1905] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:11.583055 11426 ts_tablet_manager.cc:1918] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:11.583112 11426 log.cc:1196] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/65226da7051a4c77ac5f2676d19584fa
I20260925 14:55:11.583350 11426 ts_tablet_manager.cc:1939] T 65226da7051a4c77ac5f2676d19584fa P db1a32dd268c4a70bebe6023be8c605d: Deleting consensus metadata
I20260925 14:55:11.583536 11425 ts_tablet_manager.cc:1918] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:11.583612 11425 log.cc:1196] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/65226da7051a4c77ac5f2676d19584fa
I20260925 14:55:11.583830 11425 ts_tablet_manager.cc:1939] T 65226da7051a4c77ac5f2676d19584fa P d5409261501649d998bfc81fbef36fae: Deleting consensus metadata
I20260925 14:55:11.584156 10883 catalog_manager.cc:4841] TS d5409261501649d998bfc81fbef36fae (127.9.141.65:40409): tablet 65226da7051a4c77ac5f2676d19584fa (table test-table [id=57633aa5b05c4195aa2db1a86aca8d45]) successfully deleted
I20260925 14:55:11.584174 11427 ts_tablet_manager.cc:1918] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:11.584177 10881 catalog_manager.cc:4841] TS db1a32dd268c4a70bebe6023be8c605d (127.9.141.66:40043): tablet 65226da7051a4c77ac5f2676d19584fa (table test-table [id=57633aa5b05c4195aa2db1a86aca8d45]) successfully deleted
I20260925 14:55:11.584218 11427 log.cc:1196] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/65226da7051a4c77ac5f2676d19584fa
I20260925 14:55:11.584455 11427 ts_tablet_manager.cc:1939] T 65226da7051a4c77ac5f2676d19584fa P a1130048adf04d3086521a1749776820: Deleting consensus metadata
I20260925 14:55:11.584858 10882 catalog_manager.cc:4841] TS a1130048adf04d3086521a1749776820 (127.9.141.67:36451): tablet 65226da7051a4c77ac5f2676d19584fa (table test-table [id=57633aa5b05c4195aa2db1a86aca8d45]) successfully deleted
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[10851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:11.592790 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.582628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.9.141.65:40409)
0925 14:55:11.582711 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.582713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.584516 (+ 1803us) server_negotiation.cc:407] Connection header received
0925 14:55:11.584541 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.584542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.584557 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.584611 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.585375 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.585847 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.586561 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.586752 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.586824 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.589714 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.589724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.589724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.589731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.591523 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.591927 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.591930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.591931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.591995 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.592202 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.592203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.592203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.592347 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.592550 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.592611 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.592691 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 14:55:11.593153 11088 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43544
I20260925 14:55:11.600795 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:11.594172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.9.141.126:40711)
0925 14:55:11.594224 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:11.594226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:11.594236 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:11.594309 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:11.594310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:11.594320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:11.594370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:11.594867 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.595349 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.596126 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:11.596351 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:11.596471 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:11.598157 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:11.598158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:11.598158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:11.598167 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:11.599605 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.599954 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.599955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.599956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.600012 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:11.600220 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:11.600221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:11.600222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:11.600336 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:11.600519 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:11.600621 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:55:11.600695 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 14:55:11.601123 10896 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53156
I20260925 14:55:11.601825 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10968
I20260925 14:55:11.609640 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 11108
I20260925 14:55:11.614312 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 11248
I20260925 14:55:11.620016 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 10864
2026-09-25T14:55:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2186 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[11462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[11462](info): set up 2 sockets
Sep 25 14:55:11 dist-test-slave-1sv7 krb5kdc[11462](info): commencing operation
krb5kdc: starting...
W20260925 14:55:12.664305 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:12 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:12.784017 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41117
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:41117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:12.866976 11475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:12.867213 11475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:12.867238 11475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:12.868467 11475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:12.868511 11475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:55:12.868525 11475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:12.868537 11475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:12.868551 11475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:12.868562 11475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:12.870054 11475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:41117
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:12.870307 11475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:12.870541 11475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:12.875334 11483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:12.875335 11481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:12.875427 11480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:12.875582 11475 server_base.cc:1034] running on GCE node
I20260925 14:55:12.875803 11475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:12.876083 11475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:12.877233 11475 hybrid_clock.cc:648] HybridClock initialized: now 1790348112877185 us; error 64 us; skew 500 ppm
Sep 25 14:55:12 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:12.879575 11475 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:12.880015 11475 webserver.cc:460] Webserver started at http://127.9.141.126:39831/ using document root <none> and password file <none>
I20260925 14:55:12.880241 11475 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:12.880286 11475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:12.880470 11475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:12.881771 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "04854b541b2d4ed2acb76311fd752337"
format_stamp: "Formatted at 2026-09-25 14:55:12 on dist-test-slave-1sv7"
server_key: "1da49609497735678db3db9e6e32e7f5"
server_key_iv: "3b866a767758262d93d86d291a66329f"
server_key_version: "encryptionkey@0"
I20260925 14:55:12.882130 11475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "04854b541b2d4ed2acb76311fd752337"
format_stamp: "Formatted at 2026-09-25 14:55:12 on dist-test-slave-1sv7"
server_key: "1da49609497735678db3db9e6e32e7f5"
server_key_iv: "3b866a767758262d93d86d291a66329f"
server_key_version: "encryptionkey@0"
I20260925 14:55:12.883445 11475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:12.884177 11490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:12.884423 11475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:12.884529 11475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "04854b541b2d4ed2acb76311fd752337"
format_stamp: "Formatted at 2026-09-25 14:55:12 on dist-test-slave-1sv7"
server_key: "1da49609497735678db3db9e6e32e7f5"
server_key_iv: "3b866a767758262d93d86d291a66329f"
server_key_version: "encryptionkey@0"
I20260925 14:55:12.884610 11475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:12.892678 11475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:12.894410 11475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:12.894562 11475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:12.899334 11475 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:41117
I20260925 14:55:12.899408 11552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:41117 every 8 connection(s)
I20260925 14:55:12.899734 11475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:12.900326 11553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:12.902654 11553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Bootstrap starting.
I20260925 14:55:12.903299 11553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:12.903581 11553 log.cc:824] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:12.904245 11553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: No bootstrap required, opened a new log
I20260925 14:55:12.906380 11553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04854b541b2d4ed2acb76311fd752337" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41117 } }
I20260925 14:55:12.906540 11553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:12.906625 11553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04854b541b2d4ed2acb76311fd752337, State: Initialized, Role: FOLLOWER
I20260925 14:55:12.906814 11553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04854b541b2d4ed2acb76311fd752337" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41117 } }
I20260925 14:55:12.906916 11553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:12.906961 11553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:12.907008 11553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:12.907518 11553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04854b541b2d4ed2acb76311fd752337" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41117 } }
I20260925 14:55:12.907636 11553 leader_election.cc:302] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04854b541b2d4ed2acb76311fd752337; no voters:
I20260925 14:55:12.907836 11553 leader_election.cc:288] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:12.907891 11556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:12.908089 11553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:12.908115 11556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [term 1 LEADER]: Becoming Leader. State: Replica: 04854b541b2d4ed2acb76311fd752337, State: Running, Role: LEADER
I20260925 14:55:12.908295 11556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04854b541b2d4ed2acb76311fd752337" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41117 } }
I20260925 14:55:12.908676 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 11475
I20260925 14:55:12.908687 11557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04854b541b2d4ed2acb76311fd752337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04854b541b2d4ed2acb76311fd752337" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41117 } } }
I20260925 14:55:12.908761 11557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:12.908773 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:12.908968 9781 external_mini_cluster.cc:1428] Setting key 378ebc23635d1f4da799f1b44418cddf
I20260925 14:55:12.908965 11558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04854b541b2d4ed2acb76311fd752337. Latest consensus state: current_term: 1 leader_uuid: "04854b541b2d4ed2acb76311fd752337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04854b541b2d4ed2acb76311fd752337" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41117 } } }
I20260925 14:55:12.909283 11564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:12.909318 11558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:12.909885 11564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:12.912349 11564 catalog_manager.cc:1347] Generated new cluster ID: d00fe5c2ca484a35a540ae5221492d9b
I20260925 14:55:12.912447 11564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:12 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:12.920542 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:12.909596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.9.141.126:41117)
0925 14:55:12.909901 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:12.909911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:12.909944 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:55:12.910057 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:12.910063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:12.910119 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:12.910254 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:12.911021 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:12.912195 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:12.913284 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:12.913970 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:12.914086 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:12.916900 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:12.916904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:12.916904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:12.916916 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:12.918842 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:12.919240 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:12.919242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:12.919243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:12.919356 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:12.919565 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:12.919566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:12.919566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:12.919859 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:12.920015 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:12.920149 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:55:12.920280 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":258,"thread_start_us":70,"threads_started":1}
I20260925 14:55:12.927261 11564 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:12.927794 11564 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:12.935000 11564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Generated new TSK 0
I20260925 14:55:12.935191 11564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:12.964313 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41117
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:13.064709 11579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.064915 11579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.064939 11579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.065995 11579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.066028 11579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.066048 11579 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:55:13.066061 11579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:13.066076 11579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:13.067440 11579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:41117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:13.067618 11579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.067794 11579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.072898 11587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.072928 11584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.072949 11585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.072981 11579 server_base.cc:1034] running on GCE node
I20260925 14:55:13.073340 11579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.073572 11579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.074717 11579 hybrid_clock.cc:648] HybridClock initialized: now 1790348113074695 us; error 38 us; skew 500 ppm
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.077250 11579 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:13.077623 11579 webserver.cc:460] Webserver started at http://127.9.141.65:43247/ using document root <none> and password file <none>
I20260925 14:55:13.077797 11579 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.077837 11579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.077924 11579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:13.078912 11579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "317b3172f04f43629f126edce41068da"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "686c53724520d4f6b2ab0e787661bfc2"
server_key_iv: "4b6723208d3cc7494eaf6e9f47dcdc7b"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.079208 11579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "317b3172f04f43629f126edce41068da"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "686c53724520d4f6b2ab0e787661bfc2"
server_key_iv: "4b6723208d3cc7494eaf6e9f47dcdc7b"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.080533 11579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:13.081269 11594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.081449 11579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:13.081548 11579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "317b3172f04f43629f126edce41068da"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "686c53724520d4f6b2ab0e787661bfc2"
server_key_iv: "4b6723208d3cc7494eaf6e9f47dcdc7b"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.081631 11579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.099907 11579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.102305 11579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.102478 11579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.103148 11579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:13.103202 11579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.103237 11579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:13.103294 11579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:13.110515 11579 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:37607
I20260925 14:55:13.110586 11709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:37607 every 8 connection(s)
I20260925 14:55:13.110881 11579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:13.113811 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.103952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:38167 (local address 127.9.141.126:41117)
0925 14:55:13.104013 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.104016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.104234 (+ 218us) server_negotiation.cc:407] Connection header received
0925 14:55:13.104872 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.104874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.104889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.104942 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.105573 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.106102 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.107135 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.107335 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.107414 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.110830 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.110832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.110832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.110843 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.112353 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.112807 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.112808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.112808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.112878 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.113138 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.113139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.113139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.113257 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.113332 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.113617 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.113719 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:55:13.113982 11603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.103886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.9.141.65:38167)
0925 14:55:13.104089 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.104127 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.104744 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.105022 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.105026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.105274 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.105470 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.105474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.106209 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.106210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.107037 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.107040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.107421 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.107422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.107557 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.108089 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.108100 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.110710 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.112495 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.112502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.112504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.112725 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.112959 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.112960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.112960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.113070 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.113338 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.113345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.113486 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.113594 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.113749 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260925 14:55:13.114542 11710 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41117
I20260925 14:55:13.114641 11710 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:13.114836 11710 heartbeater.cc:511] Master 127.9.141.126:41117 requested a full tablet report, sending...
I20260925 14:55:13.115408 11506 ts_manager.cc:194] Registered new tserver with Master: 317b3172f04f43629f126edce41068da (127.9.141.65:37607)
I20260925 14:55:13.116160 11506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:38167
I20260925 14:55:13.120184 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 11579
I20260925 14:55:13.120288 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:13.120507 9781 external_mini_cluster.cc:1428] Setting key 424679586f0afedc988124525c4b95e8
I20260925 14:55:13.121052 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.115576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:52963 (local address 127.9.141.126:41117)
0925 14:55:13.115611 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.115613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.115801 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:55:13.115833 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.115834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.115844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.115911 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.116180 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.116913 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.117600 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.117784 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.117840 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.118710 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.118711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.118712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.118717 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.119864 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.120189 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.120190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.120190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.120253 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.120503 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.120504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.120504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.120638 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.120707 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.120892 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.120966 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:55:13.121057 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.115540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.9.141.65:52963)
0925 14:55:13.115717 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.115727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.115797 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.115914 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.115915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.116066 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.116115 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.116116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.117016 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.117017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.117527 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.117529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.117853 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.117854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.117909 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.118358 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.118365 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.118633 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.119935 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.119936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.119936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.120088 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.120324 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.120325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.120325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.120426 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.120735 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.120736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.120780 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.120852 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.120958 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:13.148170 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41117
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:13.229362 11719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.229560 11719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.229585 11719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.230728 11719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.230777 11719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.230798 11719 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:55:13.230809 11719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:13.230826 11719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:13.232270 11719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:41117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:13.232535 11719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.232789 11719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.237736 11727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.237741 11725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.237866 11724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.237852 11719 server_base.cc:1034] running on GCE node
I20260925 14:55:13.238193 11719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.238441 11719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.239606 11719 hybrid_clock.cc:648] HybridClock initialized: now 1790348113239595 us; error 30 us; skew 500 ppm
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.242070 11719 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:13.242439 11719 webserver.cc:460] Webserver started at http://127.9.141.66:34439/ using document root <none> and password file <none>
I20260925 14:55:13.242661 11719 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.242707 11719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.242877 11719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:13.243927 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "508074b997654a80863c999608271c26"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "063d79312f2de953d29dfb00a5ced3e8"
server_key_iv: "2d39b2467b93a729a956ddf449f6254e"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.244277 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "508074b997654a80863c999608271c26"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "063d79312f2de953d29dfb00a5ced3e8"
server_key_iv: "2d39b2467b93a729a956ddf449f6254e"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.245695 11719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:13.246497 11734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.246717 11719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:13.246785 11719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "508074b997654a80863c999608271c26"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "063d79312f2de953d29dfb00a5ced3e8"
server_key_iv: "2d39b2467b93a729a956ddf449f6254e"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.246894 11719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.254733 11719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.257021 11719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.257164 11719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.257670 11719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:13.257732 11719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.257781 11719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:13.257802 11719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.264739 11719 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:42163
I20260925 14:55:13.264848 11849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:42163 every 8 connection(s)
I20260925 14:55:13.265138 11719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:13.269165 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.258227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:46657 (local address 127.9.141.126:41117)
0925 14:55:13.258309 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.258311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.258670 (+ 359us) server_negotiation.cc:407] Connection header received
0925 14:55:13.259356 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.259358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.259372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.259436 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.260362 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.261196 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.262842 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.263060 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.263132 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.265860 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.265861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.265862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.265869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.267548 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.268021 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.268022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.268023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.268087 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.268402 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.268403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.268404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.268528 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.268614 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.268948 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.269037 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:55:13.269287 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.258334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.9.141.66:46657)
0925 14:55:13.258526 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.258562 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.259224 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.259551 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.259556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.259954 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.260253 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.260259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.261293 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.261294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.262746 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.262748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.263154 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.263155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.263331 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.263918 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.263929 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.265744 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.267666 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.267673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.267676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.267924 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.268162 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.268163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.268164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.268285 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.268605 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.268611 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.268761 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.268861 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.269052 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260925 14:55:13.269842 11850 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41117
I20260925 14:55:13.269925 11850 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:13.270126 11850 heartbeater.cc:511] Master 127.9.141.126:41117 requested a full tablet report, sending...
I20260925 14:55:13.270517 11507 ts_manager.cc:194] Registered new tserver with Master: 508074b997654a80863c999608271c26 (127.9.141.66:42163)
I20260925 14:55:13.270911 11507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:46657
I20260925 14:55:13.273386 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 11719
I20260925 14:55:13.273465 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:13.273660 9781 external_mini_cluster.cc:1428] Setting key 2c17531b0507c379f8b7d12a8fe4f9c2
I20260925 14:55:13.276633 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.271020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:38189 (local address 127.9.141.126:41117)
0925 14:55:13.271108 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.271110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.271156 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:55:13.271230 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.271230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.271241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.271290 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.271565 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.272158 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.272909 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.273105 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.273188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.274043 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.274045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.274045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.274052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.275479 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.275894 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.275896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.275896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.275948 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.276150 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.276150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.276151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.276281 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.276340 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.276509 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.276571 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:55:13.276707 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.270934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.9.141.66:38189)
0925 14:55:13.271100 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.271110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.271171 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.271303 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.271303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.271450 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.271496 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.271497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.272244 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.272245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.272796 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.272797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.273170 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.273171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.273221 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.273637 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.273648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.273961 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.275576 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.275577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.275577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.275796 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.276020 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.276021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.276021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.276088 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.276345 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.276347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.276415 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.276488 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.276592 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:13.298847 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41117
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:13.380247 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:13.380532 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:13.380563 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:13.381683 11859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:13.381726 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:13.381747 11859 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:55:13.381758 11859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:13.381775 11859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:13.383268 11859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:41117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:13.383497 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:13.383735 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:13.389010 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.389030 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:13.389015 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:13.389063 11859 server_base.cc:1034] running on GCE node
I20260925 14:55:13.389389 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:13.389585 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:13.390759 11859 hybrid_clock.cc:648] HybridClock initialized: now 1790348113390745 us; error 28 us; skew 500 ppm
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:13.393486 11859 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:13.393922 11859 webserver.cc:460] Webserver started at http://127.9.141.67:36769/ using document root <none> and password file <none>
I20260925 14:55:13.394168 11859 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:13.394214 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:13.394351 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:13.395421 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "77bab1bee7354d2fab28e2d8df23c472"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "f641016ff191813ce066672a9fe5e25f"
server_key_iv: "adbc3effad596f90d84507e0f4a7d549"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.395776 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "77bab1bee7354d2fab28e2d8df23c472"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "f641016ff191813ce066672a9fe5e25f"
server_key_iv: "adbc3effad596f90d84507e0f4a7d549"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.397177 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:13.397954 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.398151 11859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:13.398280 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "77bab1bee7354d2fab28e2d8df23c472"
format_stamp: "Formatted at 2026-09-25 14:55:13 on dist-test-slave-1sv7"
server_key: "f641016ff191813ce066672a9fe5e25f"
server_key_iv: "adbc3effad596f90d84507e0f4a7d549"
server_key_version: "encryptionkey@0"
I20260925 14:55:13.398360 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:13.409655 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:13.411980 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:13.412144 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:13.412709 11859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:13.412768 11859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.412823 11859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:13.412855 11859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.420106 11859 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:43743
I20260925 14:55:13.420493 11989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:43743 every 8 connection(s)
I20260925 14:55:13.420568 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:13.423709 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 11859
I20260925 14:55:13.423800 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:13.423978 9781 external_mini_cluster.cc:1428] Setting key dc6b2b45dbbbab16ca4c4d00b5cfc875
I20260925 14:55:13.425267 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.413322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:54321 (local address 127.9.141.126:41117)
0925 14:55:13.413384 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.413386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.413795 (+ 409us) server_negotiation.cc:407] Connection header received
0925 14:55:13.414513 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.414514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.414530 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.414666 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.415619 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.416478 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.418172 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.418451 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.418515 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.421326 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.421327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.421328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.421335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.423124 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.423565 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.423567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.423567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.423638 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.424074 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.424075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.424076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.424692 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.424786 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.425073 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.425166 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:55:13.425493 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.413318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.9.141.67:54321)
0925 14:55:13.413586 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.413628 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.414356 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.414762 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.414768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.415172 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.415487 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.415494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.416597 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.416598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.418047 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.418050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.418549 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.418550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.418709 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.419348 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.419360 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.421222 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.423222 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.423227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.423229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.423471 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.423736 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.423737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.423737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.423977 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.424775 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.424782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.424932 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.425049 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.425256 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":84,"threads_started":1}
I20260925 14:55:13.425980 11990 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41117
I20260925 14:55:13.426081 11990 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:13.426266 11990 heartbeater.cc:511] Master 127.9.141.126:41117 requested a full tablet report, sending...
I20260925 14:55:13.426661 11506 ts_manager.cc:194] Registered new tserver with Master: 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.427078 11506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:54321
I20260925 14:55:13.432752 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.427092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:41099 (local address 127.9.141.126:41117)
0925 14:55:13.427136 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.427138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.427241 (+ 103us) server_negotiation.cc:407] Connection header received
0925 14:55:13.427305 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.427306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.427316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.427371 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.427684 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.428155 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.428937 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.429137 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.429216 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.430147 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.430148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.430148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.430155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.431538 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.431970 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.431971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.431972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.432028 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.432223 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.432224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.432224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.432334 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.432429 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.432580 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.432656 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:13.432757 11996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.427036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.9.141.67:41099)
0925 14:55:13.427187 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.427197 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.427254 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.427386 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.427387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.427559 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.427608 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.427609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.428252 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.428253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.428845 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.428847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.429218 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.429219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.429293 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.429759 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.429764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.430041 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.431658 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.431717 (+ 59us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.431717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.431889 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.432102 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.432102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.432103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.432159 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.432419 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.432419 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.432466 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.432556 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.432656 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260925 14:55:13.436352 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:13.459679 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.452484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.9.141.126:41117)
0925 14:55:13.452550 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.452552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.452561 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:13.452604 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.452605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.452613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.452655 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.453278 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.453923 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.454657 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.454850 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.454908 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.457267 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.457268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.457268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.457275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.458629 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.459010 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.459011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.459011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.459059 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.459225 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.459225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.459226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.459324 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.459399 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.459517 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.459579 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:55:13.461470 11506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:13.463063 11506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:13.470082 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.465791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.0.0.1:44984)
0925 14:55:13.465935 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.465968 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.466023 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.466617 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.466620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.466623 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.466694 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.466696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.467632 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.467633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.468841 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.468843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.469771 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.469772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.469851 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.469866 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.469944 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.470001 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260925 14:55:13.470283 12009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.465791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:42163 (local address 127.0.0.1:56828)
0925 14:55:13.466118 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.466127 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.466169 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.466630 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.466630 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.466633 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.466713 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.466715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.468060 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.468061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.469072 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.469074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.470019 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.470020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.470083 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.470087 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.470152 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.470189 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260925 14:55:13.470311 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.465980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.9.141.66:42163)
0925 14:55:13.466246 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.466251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.466263 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:13.466308 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.466310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.466327 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.466824 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.466888 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.467953 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.469159 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.469885 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.469990 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.470047 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.470167 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.470214 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
I20260925 14:55:13.470465 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.466224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.9.141.67:43743)
0925 14:55:13.466480 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.466483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.466493 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:13.466529 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.466531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.466545 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.466622 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.466801 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.467958 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.469225 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.469993 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.470060 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.470115 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.470320 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.470364 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":144,"threads_started":1}
I20260925 14:55:13.470570 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.466191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.9.141.65:37607)
0925 14:55:13.466420 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.466425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.466436 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:13.466478 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.466480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.466497 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.466807 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.466870 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.467534 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.468931 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.469680 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.469757 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.469819 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.470401 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.470444 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260925 14:55:13.470620 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.465984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:43743 (local address 127.0.0.1:55424)
0925 14:55:13.466361 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.466371 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.466418 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.466682 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.466683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.466685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.466718 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.466720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.468044 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.468045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.469161 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.469163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.470156 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.470157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.470205 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.470207 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.470253 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.470281 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":24,"thread_start_us":80,"threads_started":1}
I20260925 14:55:13.471138 11784 tablet_service.cc:1467] Processing CreateTablet for tablet b0f456e8d85f494585268f38dc24b5c1 (DEFAULT_TABLE table=test-table [id=4ac3c887b0b94919b4ee25335be1cbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:13.471136 11924 tablet_service.cc:1467] Processing CreateTablet for tablet b0f456e8d85f494585268f38dc24b5c1 (DEFAULT_TABLE table=test-table [id=4ac3c887b0b94919b4ee25335be1cbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:13.471487 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f456e8d85f494585268f38dc24b5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.471487 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f456e8d85f494585268f38dc24b5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.471504 11644 tablet_service.cc:1467] Processing CreateTablet for tablet b0f456e8d85f494585268f38dc24b5c1 (DEFAULT_TABLE table=test-table [id=4ac3c887b0b94919b4ee25335be1cbec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:13.471853 11644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0f456e8d85f494585268f38dc24b5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.474735 12013 tablet_bootstrap.cc:492] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Bootstrap starting.
I20260925 14:55:13.474929 12014 tablet_bootstrap.cc:492] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Bootstrap starting.
I20260925 14:55:13.475545 12013 tablet_bootstrap.cc:654] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.475862 12014 tablet_bootstrap.cc:654] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.475893 12013 log.cc:824] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:13.476190 12014 log.cc:824] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:13.476240 12015 tablet_bootstrap.cc:492] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Bootstrap starting.
I20260925 14:55:13.476687 12013 tablet_bootstrap.cc:492] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: No bootstrap required, opened a new log
I20260925 14:55:13.476761 12013 ts_tablet_manager.cc:1397] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:13.476909 12014 tablet_bootstrap.cc:492] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: No bootstrap required, opened a new log
I20260925 14:55:13.476970 12014 ts_tablet_manager.cc:1397] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:13.478811 12013 raft_consensus.cc:348] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.478855 12014 raft_consensus.cc:348] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.478993 12013 raft_consensus.cc:374] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.478993 12014 raft_consensus.cc:374] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.479028 12013 raft_consensus.cc:729] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317b3172f04f43629f126edce41068da, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.479028 12014 raft_consensus.cc:729] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bab1bee7354d2fab28e2d8df23c472, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.479148 12013 consensus_queue.cc:260] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.479171 12014 consensus_queue.cc:260] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.479429 12013 ts_tablet_manager.cc:1428] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:55:13.479429 12014 ts_tablet_manager.cc:1428] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:55:13.479471 11710 heartbeater.cc:503] Master 127.9.141.126:41117 was elected leader, sending a full tablet report...
I20260925 14:55:13.479573 11990 heartbeater.cc:503] Master 127.9.141.126:41117 was elected leader, sending a full tablet report...
I20260925 14:55:13.480077 12015 tablet_bootstrap.cc:654] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.480453 12015 log.cc:824] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:13.481235 12015 tablet_bootstrap.cc:492] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: No bootstrap required, opened a new log
I20260925 14:55:13.481417 12015 ts_tablet_manager.cc:1397] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 14:55:13.482914 12015 raft_consensus.cc:348] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.483078 12015 raft_consensus.cc:374] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.483124 12015 raft_consensus.cc:729] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508074b997654a80863c999608271c26, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.483259 12015 consensus_queue.cc:260] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.483541 12015 ts_tablet_manager.cc:1428] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:55:13.483604 11850 heartbeater.cc:503] Master 127.9.141.126:41117 was elected leader, sending a full tablet report...
I20260925 14:55:13.486613 12021 raft_consensus.cc:482] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:13.486708 12021 raft_consensus.cc:504] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.487099 12021 leader_election.cc:288] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 317b3172f04f43629f126edce41068da (127.9.141.65:37607), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.490286 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.487727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:57969 (local address 127.9.141.67:43743)
0925 14:55:13.487833 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.487835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.487843 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:13.487863 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.487864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.487871 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.487948 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.488101 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.488631 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.489567 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.489925 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.490019 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.490028 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.490182 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.490215 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 14:55:13.490353 12022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.487449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:43743 (local address 127.9.141.66:57969)
0925 14:55:13.487595 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.487602 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.487636 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.487953 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.487954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.487956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.488021 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.488023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.488725 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.488726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.489477 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.489479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.490019 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.490019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.490089 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.490093 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.490174 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.490214 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260925 14:55:13.490355 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.487332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.9.141.66:42007)
0925 14:55:13.487374 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.487386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.487441 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.487931 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.487932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.487934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.487998 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.488000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.488710 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.488711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.489467 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.489469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.490043 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.490045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.490135 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.490138 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.490212 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.490251 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":26}
I20260925 14:55:13.490322 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.487656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:42007 (local address 127.9.141.65:37607)
0925 14:55:13.487720 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.487722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.487730 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:13.487757 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.487758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.487767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.487926 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.488078 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.488634 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.489577 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.489953 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.490043 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.490051 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.490219 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.490253 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:55:13.490713 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f456e8d85f494585268f38dc24b5c1" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472" is_pre_election: true
I20260925 14:55:13.490875 11944 raft_consensus.cc:2454] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 0.
I20260925 14:55:13.490967 11664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f456e8d85f494585268f38dc24b5c1" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317b3172f04f43629f126edce41068da" is_pre_election: true
I20260925 14:55:13.491073 11735 leader_election.cc:302] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508074b997654a80863c999608271c26, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:13.491106 11664 raft_consensus.cc:2454] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 0.
I20260925 14:55:13.491194 12021 raft_consensus.cc:2781] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:13.491238 12021 raft_consensus.cc:482] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:13.491281 12021 raft_consensus.cc:3037] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.491978 12021 raft_consensus.cc:504] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.492120 12021 leader_election.cc:288] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 election: Requested vote from peers 317b3172f04f43629f126edce41068da (127.9.141.65:37607), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.492264 11664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f456e8d85f494585268f38dc24b5c1" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317b3172f04f43629f126edce41068da"
I20260925 14:55:13.492311 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0f456e8d85f494585268f38dc24b5c1" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472"
I20260925 14:55:13.492394 11944 raft_consensus.cc:3037] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.492357 11664 raft_consensus.cc:3037] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.493041 11944 raft_consensus.cc:2454] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 1.
I20260925 14:55:13.493077 11664 raft_consensus.cc:2454] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 1.
I20260925 14:55:13.493224 11735 leader_election.cc:302] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508074b997654a80863c999608271c26, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:13.493330 12021 raft_consensus.cc:2781] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:13.493482 12021 raft_consensus.cc:686] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 1 LEADER]: Becoming Leader. State: Replica: 508074b997654a80863c999608271c26, State: Running, Role: LEADER
I20260925 14:55:13.493580 12021 consensus_queue.cc:237] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.494321 11507 catalog_manager.cc:5495] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 508074b997654a80863c999608271c26 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "508074b997654a80863c999608271c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } health_report { overall_health: UNKNOWN } } }
W20260925 14:55:13.515774 11851 tablet.cc:2369] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:55:13 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348112, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:13.528096 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.521133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41117 (local address 127.0.0.1:55004)
0925 14:55:13.521302 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.521312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.521378 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.521510 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.521510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.521815 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:13.521863 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.521865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.522494 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.522495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.523023 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.523025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.523367 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.523367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.523408 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.524020 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:13.524025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:13.525477 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:13.527066 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.527072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.527074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.527262 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.527451 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:13.527451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:13.527452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:13.527504 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:13.527818 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:13.527823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:13.527860 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.527911 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.527997 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260925 14:55:13.528191 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.521228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.9.141.126:41117)
0925 14:55:13.521320 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.521327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.521378 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:55:13.521437 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.521438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.521447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.521490 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:13.521942 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.522424 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.523099 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.523301 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.523355 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.525631 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:13.525632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:13.525632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:13.525638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:13.526982 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.527338 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.527339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.527339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.527391 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:13.527585 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:13.527586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:13.527587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:13.527753 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:13.527818 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.527945 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.528045 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":66}
I20260925 14:55:13.530223 11507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:55:13.530438 11507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:13.532128 11644 tablet_service.cc:1467] Processing CreateTablet for tablet 374ff343f4254b7486d702c91c232ba1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cfbd70c7c534bd2bcc23b60ec606cc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:13.532202 11924 tablet_service.cc:1467] Processing CreateTablet for tablet 374ff343f4254b7486d702c91c232ba1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cfbd70c7c534bd2bcc23b60ec606cc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:13.532303 11644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374ff343f4254b7486d702c91c232ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.532326 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374ff343f4254b7486d702c91c232ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.533823 12013 tablet_bootstrap.cc:492] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: Bootstrap starting.
I20260925 14:55:13.533818 12014 tablet_bootstrap.cc:492] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: Bootstrap starting.
I20260925 14:55:13.534407 12013 tablet_bootstrap.cc:654] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.534719 12014 tablet_bootstrap.cc:654] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.535272 12013 tablet_bootstrap.cc:492] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: No bootstrap required, opened a new log
I20260925 14:55:13.535329 12013 ts_tablet_manager.cc:1397] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:55:13.535482 12013 raft_consensus.cc:348] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.535540 12013 raft_consensus.cc:374] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.535554 12013 raft_consensus.cc:729] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317b3172f04f43629f126edce41068da, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.535611 12013 consensus_queue.cc:260] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.535735 12013 ts_tablet_manager.cc:1428] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.535789 12019 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.535869 12019 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:13.535851 12014 tablet_bootstrap.cc:492] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: No bootstrap required, opened a new log
I20260925 14:55:13.536019 12014 ts_tablet_manager.cc:1397] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:55:13.536284 12014 raft_consensus.cc:348] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.536398 12014 raft_consensus.cc:374] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.536428 12014 raft_consensus.cc:729] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bab1bee7354d2fab28e2d8df23c472, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.536471 12014 consensus_queue.cc:260] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.536558 12014 ts_tablet_manager.cc:1428] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.536695 12018 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.536727 11784 tablet_service.cc:1467] Processing CreateTablet for tablet 374ff343f4254b7486d702c91c232ba1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1cfbd70c7c534bd2bcc23b60ec606cc7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:13.536791 12018 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:13.536871 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 374ff343f4254b7486d702c91c232ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.538497 12015 tablet_bootstrap.cc:492] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: Bootstrap starting.
I20260925 14:55:13.538933 12015 tablet_bootstrap.cc:654] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.539659 12015 tablet_bootstrap.cc:492] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: No bootstrap required, opened a new log
I20260925 14:55:13.539716 12015 ts_tablet_manager.cc:1397] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:13.539870 12015 raft_consensus.cc:348] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.539979 12015 raft_consensus.cc:374] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.540000 12015 raft_consensus.cc:729] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508074b997654a80863c999608271c26, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.540040 12015 consensus_queue.cc:260] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.540132 12015 ts_tablet_manager.cc:1428] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:13.540169 12021 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.540257 12021 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:55:13.611663 11711 tablet.cc:2369] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:13.619784 12019 raft_consensus.cc:482] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:13.619868 12019 raft_consensus.cc:504] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.620203 12019 leader_election.cc:288] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 508074b997654a80863c999608271c26 (127.9.141.66:42163), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.623385 12034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.620357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:42163 (local address 127.9.141.65:51789)
0925 14:55:13.620515 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.620528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.620588 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.620757 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.620758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.620761 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.620842 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.620845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.621612 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.621613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.622540 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.622542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.623074 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.623076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.623165 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.623170 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.623228 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.623273 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":56,"thread_start_us":77,"threads_started":1}
I20260925 14:55:13.623384 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.620493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:51789 (local address 127.9.141.66:42163)
0925 14:55:13.620567 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.620569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.620581 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:13.620667 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.620668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.620679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.620740 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.620918 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.621526 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.622630 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.622995 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.623054 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.623061 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.623246 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.623318 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:55:13.623463 12012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.620489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:49787 (local address 127.9.141.67:43743)
0925 14:55:13.620553 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.620555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.620664 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:55:13.620749 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.620750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.620760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.620818 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:13.621028 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.621526 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.622547 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.623088 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.623166 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.623174 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.623347 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.623383 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:13.623528 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.620440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:43743 (local address 127.9.141.65:49787)
0925 14:55:13.620582 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.620592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.620658 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.620864 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.620864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.620866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:13.620956 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.620958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.621647 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.621647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.622447 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.622449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.623173 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.623174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.623266 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.623270 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.623326 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.623364 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":24,"thread_start_us":73,"threads_started":1}
I20260925 14:55:13.623689 11804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374ff343f4254b7486d702c91c232ba1" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508074b997654a80863c999608271c26" is_pre_election: true
I20260925 14:55:13.623755 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374ff343f4254b7486d702c91c232ba1" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472" is_pre_election: true
I20260925 14:55:13.623812 11804 raft_consensus.cc:2454] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 0.
I20260925 14:55:13.623826 11944 raft_consensus.cc:2454] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 0.
I20260925 14:55:13.624011 11595 leader_election.cc:302] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317b3172f04f43629f126edce41068da, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:13.624154 12019 raft_consensus.cc:2781] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:13.624194 12019 raft_consensus.cc:482] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:13.624219 12019 raft_consensus.cc:3037] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.624918 12019 raft_consensus.cc:504] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.625058 12019 leader_election.cc:288] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 election: Requested vote from peers 508074b997654a80863c999608271c26 (127.9.141.66:42163), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.625201 11804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374ff343f4254b7486d702c91c232ba1" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508074b997654a80863c999608271c26"
I20260925 14:55:13.625191 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "374ff343f4254b7486d702c91c232ba1" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472"
I20260925 14:55:13.625298 11804 raft_consensus.cc:3037] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.625296 11944 raft_consensus.cc:3037] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.626039 11804 raft_consensus.cc:2454] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 1.
I20260925 14:55:13.626104 11944 raft_consensus.cc:2454] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 1.
I20260925 14:55:13.626241 11597 leader_election.cc:302] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317b3172f04f43629f126edce41068da, 508074b997654a80863c999608271c26; no voters:
I20260925 14:55:13.626343 12019 raft_consensus.cc:2781] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:13.626470 12019 raft_consensus.cc:686] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [term 1 LEADER]: Becoming Leader. State: Replica: 317b3172f04f43629f126edce41068da, State: Running, Role: LEADER
I20260925 14:55:13.626577 12019 consensus_queue.cc:237] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.627007 12036 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: TxnStatusTablet state changed. Reason: New leader 317b3172f04f43629f126edce41068da. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.627117 12036 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:13.627311 12038 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:55:13.627424 12038 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:55:13.627349 11507 catalog_manager.cc:5495] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da reported cstate change: term changed from 0 to 1, leader changed from <none> to 317b3172f04f43629f126edce41068da (127.9.141.65). New cstate: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:13.655915 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.652843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.0.0.1:44986)
0925 14:55:13.652964 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.652977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.653064 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.653332 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.653334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.653335 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:13.653391 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.653399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.654209 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.654211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.654859 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.654862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.655268 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.655269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.655347 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.655362 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.655722 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.655733 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.655778 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.655832 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 14:55:13.655915 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.652966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.9.141.65:37607)
0925 14:55:13.653039 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.653042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.653053 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:13.653211 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.653212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.653233 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.653324 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:13.653476 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.654100 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.654969 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.655182 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.655266 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.655507 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.655618 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.655723 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.655794 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.655815 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:55:13.658048 11944 raft_consensus.cc:1261] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Refusing update from remote peer 317b3172f04f43629f126edce41068da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:13.658105 11804 raft_consensus.cc:1261] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Refusing update from remote peer 317b3172f04f43629f126edce41068da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:13.658411 12036 consensus_queue.cc:1035] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [LEADER]: Connected to new peer: Peer: permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:13.658565 12019 consensus_queue.cc:1035] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:13.659758 12021 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: TxnStatusTablet state changed. Reason: New leader 317b3172f04f43629f126edce41068da. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.659858 12021 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:13.659818 12018 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: TxnStatusTablet state changed. Reason: New leader 317b3172f04f43629f126edce41068da. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.659911 12018 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:13.661093 12036 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.661181 12036 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:13.661231 12036 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.661278 12036 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 317b3172f04f43629f126edce41068da: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:13.661330 12041 mvcc.cc:205] Tried to move back new op lower bound from 7333265873540739072 to 7333265873415127040. Current Snapshot: MvccSnapshot[applied={T|T < 7333265873540739072}]
I20260925 14:55:13.661305 12021 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.661386 12021 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 508074b997654a80863c999608271c26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:13.661432 12042 mvcc.cc:205] Tried to move back new op lower bound from 7333265873540739072 to 7333265873415127040. Current Snapshot: MvccSnapshot[applied={T|T < 7333265873540739072}]
I20260925 14:55:13.662683 12018 tablet_replica.cc:439] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } } }
I20260925 14:55:13.662774 12018 tablet_replica.cc:442] T 374ff343f4254b7486d702c91c232ba1 P 77bab1bee7354d2fab28e2d8df23c472: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:13.667277 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.664753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.9.141.66:42163)
0925 14:55:13.664805 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.664807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.664859 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:55:13.664939 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.664940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.664957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.665025 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:13.665158 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.665722 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.666492 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.666699 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.666775 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.666936 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.667037 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.667148 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.667166 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.667186 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 14:55:13.671192 11991 tablet.cc:2369] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:13.672073 12011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.669748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:38523 (local address 127.9.141.65:37607)
0925 14:55:13.669843 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.669846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.669855 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:13.669877 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.669878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.669886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.669934 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:13.670035 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.670563 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.671351 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.671559 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.671609 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.671854 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.671879 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.671926 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.672010 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.672026 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:55:13.672133 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.669701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.9.141.66:38523)
0925 14:55:13.669772 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.669783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.669821 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.669935 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.669935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.669936 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:13.669988 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.669989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.670642 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.670642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.671234 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.671235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.671645 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.671646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.671731 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.671746 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.671943 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.671946 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.672007 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.672056 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:55:13.674654 11944 raft_consensus.cc:1261] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Refusing update from remote peer 508074b997654a80863c999608271c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:13.674769 11664 raft_consensus.cc:1261] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Refusing update from remote peer 508074b997654a80863c999608271c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:13.674993 12023 consensus_queue.cc:1035] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:13.675098 12021 consensus_queue.cc:1035] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:13.676703 12043 mvcc.cc:205] Tried to move back new op lower bound from 7333265873609670656 to 7333265872870354944. Current Snapshot: MvccSnapshot[applied={T|T < 7333265873609670656}]
I20260925 14:55:13.683473 12010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.680719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:37227 (local address 127.9.141.66:42163)
0925 14:55:13.680757 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:13.680759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:13.680943 (+ 184us) server_negotiation.cc:407] Connection header received
0925 14:55:13.680967 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:13.680968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:13.680978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:13.681036 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:13.681193 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.681629 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.682852 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.683025 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.683078 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.683275 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.683297 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.683330 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:13.683413 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:55:13.683428 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:13.683552 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:13.680678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:42163 (local address 127.9.141.65:37227)
0925 14:55:13.680831 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:13.680851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:13.680912 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:13.681046 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:13.681047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:13.681048 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:13.681124 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.681126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.681701 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.681701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.682759 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:13.682762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:13.683098 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:13.683099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:13.683180 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:13.683194 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:13.683354 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:13.683356 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:13.683412 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:55:13.683456 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 14:55:13.692240 11506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:13.692355 11506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:13.693099 11506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Sending DeleteTablet for 3 replicas of tablet b0f456e8d85f494585268f38dc24b5c1
I20260925 14:55:13.693374 11784 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f456e8d85f494585268f38dc24b5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:13 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:56828
I20260925 14:55:13.693503 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f456e8d85f494585268f38dc24b5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:13 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55424
I20260925 14:55:13.693583 12059 tablet_replica.cc:330] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: stopping tablet replica
I20260925 14:55:13.693562 11644 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f456e8d85f494585268f38dc24b5c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:13 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:44984
I20260925 14:55:13.693672 12059 raft_consensus.cc:2229] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:13.693776 12059 raft_consensus.cc:2258] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:13.694252 12059 ts_tablet_manager.cc:1905] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:13.694233 11506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:13.694357 11506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:13.695978 11924 tablet_service.cc:1467] Processing CreateTablet for tablet de2915e7f21044689a6c9b7b8444ae0a (DEFAULT_TABLE table=test-table [id=ae901c7e106c4a1686378b610e0645be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:13.696123 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2915e7f21044689a6c9b7b8444ae0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.696223 12059 ts_tablet_manager.cc:1918] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:13.696346 12059 log.cc:1196] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/b0f456e8d85f494585268f38dc24b5c1
I20260925 14:55:13.696451 11644 tablet_service.cc:1467] Processing CreateTablet for tablet de2915e7f21044689a6c9b7b8444ae0a (DEFAULT_TABLE table=test-table [id=ae901c7e106c4a1686378b610e0645be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:13.696558 11644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2915e7f21044689a6c9b7b8444ae0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.696790 12059 ts_tablet_manager.cc:1939] T b0f456e8d85f494585268f38dc24b5c1 P 508074b997654a80863c999608271c26: Deleting consensus metadata
I20260925 14:55:13.697692 12013 tablet_bootstrap.cc:492] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Bootstrap starting.
I20260925 14:55:13.697834 11493 catalog_manager.cc:4841] TS 508074b997654a80863c999608271c26 (127.9.141.66:42163): tablet b0f456e8d85f494585268f38dc24b5c1 (table test-table [id=4ac3c887b0b94919b4ee25335be1cbec]) successfully deleted
I20260925 14:55:13.697979 11784 tablet_service.cc:1467] Processing CreateTablet for tablet de2915e7f21044689a6c9b7b8444ae0a (DEFAULT_TABLE table=test-table [id=ae901c7e106c4a1686378b610e0645be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:13.698056 12013 tablet_bootstrap.cc:654] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.698064 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de2915e7f21044689a6c9b7b8444ae0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:13.699054 12015 tablet_bootstrap.cc:492] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Bootstrap starting.
I20260925 14:55:13.699492 12015 tablet_bootstrap.cc:654] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.700114 12015 tablet_bootstrap.cc:492] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: No bootstrap required, opened a new log
I20260925 14:55:13.700155 12015 ts_tablet_manager.cc:1397] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:13.700289 12015 raft_consensus.cc:348] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.700348 12015 raft_consensus.cc:374] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.700366 12015 raft_consensus.cc:729] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508074b997654a80863c999608271c26, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.700435 12015 consensus_queue.cc:260] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.700524 12015 ts_tablet_manager.cc:1428] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:13.700678 12014 tablet_bootstrap.cc:492] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Bootstrap starting.
I20260925 14:55:13.701097 12014 tablet_bootstrap.cc:654] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:13.701979 12061 tablet_replica.cc:330] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: stopping tablet replica
I20260925 14:55:13.702106 12061 raft_consensus.cc:2229] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:13.702168 12061 raft_consensus.cc:2258] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:13.702312 12060 tablet_replica.cc:330] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: stopping tablet replica
I20260925 14:55:13.702414 12060 raft_consensus.cc:2229] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:13.702481 12060 raft_consensus.cc:2258] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:13.702880 12061 ts_tablet_manager.cc:1905] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:13.704484 12061 ts_tablet_manager.cc:1918] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:13.704602 12061 log.cc:1196] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/b0f456e8d85f494585268f38dc24b5c1
I20260925 14:55:13.704660 12013 tablet_bootstrap.cc:492] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: No bootstrap required, opened a new log
I20260925 14:55:13.704707 12013 ts_tablet_manager.cc:1397] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 14:55:13.704773 12014 tablet_bootstrap.cc:492] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: No bootstrap required, opened a new log
I20260925 14:55:13.704835 12014 ts_tablet_manager.cc:1397] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 14:55:13.704890 12013 raft_consensus.cc:348] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.704977 12013 raft_consensus.cc:374] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.705005 12013 raft_consensus.cc:729] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317b3172f04f43629f126edce41068da, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.704986 12014 raft_consensus.cc:348] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.705061 12014 raft_consensus.cc:374] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:13.705056 12013 consensus_queue.cc:260] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.705088 12014 raft_consensus.cc:729] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bab1bee7354d2fab28e2d8df23c472, State: Initialized, Role: FOLLOWER
I20260925 14:55:13.705152 12013 ts_tablet_manager.cc:1428] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:13.705144 12014 consensus_queue.cc:260] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.705233 12014 ts_tablet_manager.cc:1428] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:13.705376 12061 ts_tablet_manager.cc:1939] T b0f456e8d85f494585268f38dc24b5c1 P 317b3172f04f43629f126edce41068da: Deleting consensus metadata
I20260925 14:55:13.706009 11491 catalog_manager.cc:4841] TS 317b3172f04f43629f126edce41068da (127.9.141.65:37607): tablet b0f456e8d85f494585268f38dc24b5c1 (table test-table [id=4ac3c887b0b94919b4ee25335be1cbec]) successfully deleted
I20260925 14:55:13.706108 12060 ts_tablet_manager.cc:1905] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:13.707383 12060 ts_tablet_manager.cc:1918] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:13.707468 12060 log.cc:1196] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/b0f456e8d85f494585268f38dc24b5c1
I20260925 14:55:13.707690 12060 ts_tablet_manager.cc:1939] T b0f456e8d85f494585268f38dc24b5c1 P 77bab1bee7354d2fab28e2d8df23c472: Deleting consensus metadata
I20260925 14:55:13.708086 11491 catalog_manager.cc:4841] TS 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743): tablet b0f456e8d85f494585268f38dc24b5c1 (table test-table [id=4ac3c887b0b94919b4ee25335be1cbec]) successfully deleted
I20260925 14:55:13.816617 12021 raft_consensus.cc:482] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:13.816751 12021 raft_consensus.cc:504] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.816941 12021 leader_election.cc:288] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 317b3172f04f43629f126edce41068da (127.9.141.65:37607), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.817162 11664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2915e7f21044689a6c9b7b8444ae0a" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317b3172f04f43629f126edce41068da" is_pre_election: true
I20260925 14:55:13.817202 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2915e7f21044689a6c9b7b8444ae0a" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472" is_pre_election: true
I20260925 14:55:13.817301 11944 raft_consensus.cc:2454] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 0.
I20260925 14:55:13.817302 11664 raft_consensus.cc:2454] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 0.
I20260925 14:55:13.817467 11735 leader_election.cc:302] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508074b997654a80863c999608271c26, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:13.817566 12021 raft_consensus.cc:2781] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:13.817610 12021 raft_consensus.cc:482] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:13.817629 12021 raft_consensus.cc:3037] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.818331 12021 raft_consensus.cc:504] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.818461 12021 leader_election.cc:288] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 election: Requested vote from peers 317b3172f04f43629f126edce41068da (127.9.141.65:37607), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:13.818619 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2915e7f21044689a6c9b7b8444ae0a" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472"
I20260925 14:55:13.818632 11664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de2915e7f21044689a6c9b7b8444ae0a" candidate_uuid: "508074b997654a80863c999608271c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317b3172f04f43629f126edce41068da"
I20260925 14:55:13.818718 11664 raft_consensus.cc:3037] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.818718 11944 raft_consensus.cc:3037] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:13.819286 11944 raft_consensus.cc:2454] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 1.
I20260925 14:55:13.819470 11735 leader_election.cc:302] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 508074b997654a80863c999608271c26, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:13.819545 11664 raft_consensus.cc:2454] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 508074b997654a80863c999608271c26 in term 1.
I20260925 14:55:13.819581 12021 raft_consensus.cc:2781] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:13.819626 12021 raft_consensus.cc:686] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 1 LEADER]: Becoming Leader. State: Replica: 508074b997654a80863c999608271c26, State: Running, Role: LEADER
I20260925 14:55:13.819664 12021 consensus_queue.cc:237] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:13.820443 11506 catalog_manager.cc:5495] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 508074b997654a80863c999608271c26 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "508074b997654a80863c999608271c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:13.850518 11944 raft_consensus.cc:1261] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Refusing update from remote peer 508074b997654a80863c999608271c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:13.850518 11664 raft_consensus.cc:1261] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Refusing update from remote peer 508074b997654a80863c999608271c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:13.850800 12023 consensus_queue.cc:1035] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:13.850940 12021 consensus_queue.cc:1035] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:55:13.855394 12042 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:55:15.372968 11506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:15.373070 11506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:15.373911 11506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Sending DeleteTablet for 3 replicas of tablet de2915e7f21044689a6c9b7b8444ae0a
I20260925 14:55:15.374207 11784 tablet_service.cc:1514] Processing DeleteTablet for tablet de2915e7f21044689a6c9b7b8444ae0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:15 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:56828
I20260925 14:55:15.374219 11643 tablet_service.cc:1514] Processing DeleteTablet for tablet de2915e7f21044689a6c9b7b8444ae0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:15 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:44984
I20260925 14:55:15.374266 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet de2915e7f21044689a6c9b7b8444ae0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:15 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55424
I20260925 14:55:15.376487 12089 tablet_replica.cc:330] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: stopping tablet replica
I20260925 14:55:15.376605 12089 raft_consensus.cc:2229] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:15.376669 12089 raft_consensus.cc:2258] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:15.376788 12091 tablet_replica.cc:330] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: stopping tablet replica
I20260925 14:55:15.376849 12091 raft_consensus.cc:2229] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:15.376895 12091 raft_consensus.cc:2258] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:15.376912 12089 ts_tablet_manager.cc:1905] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:15.376997 12090 tablet_replica.cc:330] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: stopping tablet replica
I20260925 14:55:15.377054 12090 raft_consensus.cc:2229] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:15.377115 12090 raft_consensus.cc:2258] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:15.377333 12091 ts_tablet_manager.cc:1905] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:15.378059 12090 ts_tablet_manager.cc:1905] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:15.378700 12089 ts_tablet_manager.cc:1918] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:55:15.378763 12089 log.cc:1196] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/de2915e7f21044689a6c9b7b8444ae0a
I20260925 14:55:15.378870 12091 ts_tablet_manager.cc:1918] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:55:15.378927 12091 log.cc:1196] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/de2915e7f21044689a6c9b7b8444ae0a
I20260925 14:55:15.379024 12089 ts_tablet_manager.cc:1939] T de2915e7f21044689a6c9b7b8444ae0a P 508074b997654a80863c999608271c26: Deleting consensus metadata
I20260925 14:55:15.379213 12091 ts_tablet_manager.cc:1939] T de2915e7f21044689a6c9b7b8444ae0a P 77bab1bee7354d2fab28e2d8df23c472: Deleting consensus metadata
I20260925 14:55:15.379552 12090 ts_tablet_manager.cc:1918] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:55:15.379540 11493 catalog_manager.cc:4841] TS 508074b997654a80863c999608271c26 (127.9.141.66:42163): tablet de2915e7f21044689a6c9b7b8444ae0a (table test-table [id=ae901c7e106c4a1686378b610e0645be]) successfully deleted
I20260925 14:55:15.379616 12090 log.cc:1196] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/de2915e7f21044689a6c9b7b8444ae0a
I20260925 14:55:15.379540 11491 catalog_manager.cc:4841] TS 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743): tablet de2915e7f21044689a6c9b7b8444ae0a (table test-table [id=ae901c7e106c4a1686378b610e0645be]) successfully deleted
I20260925 14:55:15.379817 12090 ts_tablet_manager.cc:1939] T de2915e7f21044689a6c9b7b8444ae0a P 317b3172f04f43629f126edce41068da: Deleting consensus metadata
I20260925 14:55:15.380340 11491 catalog_manager.cc:4841] TS 317b3172f04f43629f126edce41068da (127.9.141.65:37607): tablet de2915e7f21044689a6c9b7b8444ae0a (table test-table [id=ae901c7e106c4a1686378b610e0645be]) successfully deleted
Sep 25 14:55:15 dist-test-slave-1sv7 krb5kdc[11462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348115, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:55:15 dist-test-slave-1sv7 krb5kdc[11462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348115, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:15.403439 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.391885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55010 (local address 127.9.141.126:41117)
0925 14:55:15.392053 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.392056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.392065 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:15.392095 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.392095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.392105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.392175 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:15.392820 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.393333 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.394299 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.394573 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.394641 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.400227 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:15.400228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:15.400228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:15.400236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:15.402005 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:15.402491 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:15.402493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:15.402493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:15.402570 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:15.402794 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:15.402795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:15.402795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:15.402923 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:15.403045 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.403155 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.403249 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260925 14:55:15.405200 11506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:15.405390 11506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:15.406888 11643 tablet_service.cc:1467] Processing CreateTablet for tablet 4195a89e8ddf4d09af52382c2585a2af (DEFAULT_TABLE table=test-table [id=93b453cfb91b4d8889a832c12aa5fadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:15.406926 11924 tablet_service.cc:1467] Processing CreateTablet for tablet 4195a89e8ddf4d09af52382c2585a2af (DEFAULT_TABLE table=test-table [id=93b453cfb91b4d8889a832c12aa5fadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:15.406944 11784 tablet_service.cc:1467] Processing CreateTablet for tablet 4195a89e8ddf4d09af52382c2585a2af (DEFAULT_TABLE table=test-table [id=93b453cfb91b4d8889a832c12aa5fadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:15.407058 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195a89e8ddf4d09af52382c2585a2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:15.407058 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195a89e8ddf4d09af52382c2585a2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:15.407058 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195a89e8ddf4d09af52382c2585a2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:15.408643 12101 tablet_bootstrap.cc:492] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Bootstrap starting.
I20260925 14:55:15.408659 12100 tablet_bootstrap.cc:492] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Bootstrap starting.
I20260925 14:55:15.408684 12102 tablet_bootstrap.cc:492] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Bootstrap starting.
I20260925 14:55:15.409091 12102 tablet_bootstrap.cc:654] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:15.409165 12100 tablet_bootstrap.cc:654] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:15.409242 12101 tablet_bootstrap.cc:654] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:15.409758 12102 tablet_bootstrap.cc:492] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: No bootstrap required, opened a new log
I20260925 14:55:15.409832 12102 ts_tablet_manager.cc:1397] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:15.409941 12100 tablet_bootstrap.cc:492] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: No bootstrap required, opened a new log
I20260925 14:55:15.409981 12100 ts_tablet_manager.cc:1397] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:15.409977 12102 raft_consensus.cc:348] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.410041 12102 raft_consensus.cc:374] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:15.410063 12102 raft_consensus.cc:729] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508074b997654a80863c999608271c26, State: Initialized, Role: FOLLOWER
I20260925 14:55:15.410085 12100 raft_consensus.cc:348] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.410133 12100 raft_consensus.cc:374] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:15.410147 12100 raft_consensus.cc:729] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bab1bee7354d2fab28e2d8df23c472, State: Initialized, Role: FOLLOWER
I20260925 14:55:15.410123 12102 consensus_queue.cc:260] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.410183 12100 consensus_queue.cc:260] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.410221 12102 ts_tablet_manager.cc:1428] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:15.410363 12100 ts_tablet_manager.cc:1428] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:15.410604 12101 tablet_bootstrap.cc:492] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: No bootstrap required, opened a new log
I20260925 14:55:15.410665 12101 ts_tablet_manager.cc:1397] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:15.410831 12101 raft_consensus.cc:348] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.410951 12101 raft_consensus.cc:374] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:15.411016 12101 raft_consensus.cc:729] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317b3172f04f43629f126edce41068da, State: Initialized, Role: FOLLOWER
I20260925 14:55:15.411110 12101 consensus_queue.cc:260] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.411234 12101 ts_tablet_manager.cc:1428] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:15.419531 12106 raft_consensus.cc:482] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:15.419649 12106 raft_consensus.cc:504] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.419999 12106 leader_election.cc:288] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 317b3172f04f43629f126edce41068da (127.9.141.65:37607), 508074b997654a80863c999608271c26 (127.9.141.66:42163)
I20260925 14:55:15.423516 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.420130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.9.141.67:42571)
0925 14:55:15.420345 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:15.420357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:15.420443 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:15.420620 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:15.420621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:15.420624 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:15.420691 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.420693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.421662 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.421662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.422500 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.422503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.423268 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.423268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.423323 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.423327 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:15.423373 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:55:15.423404 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":47,"thread_start_us":97,"threads_started":1}
I20260925 14:55:15.423527 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.420177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:42571 (local address 127.9.141.65:37607)
0925 14:55:15.420482 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.420484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.420496 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:15.420522 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.420523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.420536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.420605 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:15.420766 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.421550 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.422604 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.423189 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.423264 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.423274 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.423400 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.423449 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":96,"threads_started":1}
I20260925 14:55:15.423753 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.420182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:42163 (local address 127.9.141.67:50175)
0925 14:55:15.420491 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:15.420501 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:15.420541 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:15.420686 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:15.420688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:15.420690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:15.420759 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.420761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.421636 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.421637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.422730 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.422733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.423471 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.423471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.423542 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.423546 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:15.423615 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:55:15.423654 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":31,"thread_start_us":83,"threads_started":1}
I20260925 14:55:15.423774 11664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195a89e8ddf4d09af52382c2585a2af" candidate_uuid: "77bab1bee7354d2fab28e2d8df23c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317b3172f04f43629f126edce41068da" is_pre_election: true
I20260925 14:55:15.423859 11664 raft_consensus.cc:2454] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bab1bee7354d2fab28e2d8df23c472 in term 0.
I20260925 14:55:15.423986 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.420335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:50175 (local address 127.9.141.66:42163)
0925 14:55:15.420565 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.420567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.420575 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:15.420598 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.420599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.420609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.420669 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:15.420846 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.421545 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.422814 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.423387 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.423458 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.423466 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.423854 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.423903 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":98,"threads_started":1}
I20260925 14:55:15.424086 11875 leader_election.cc:302] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317b3172f04f43629f126edce41068da, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:15.424155 11804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195a89e8ddf4d09af52382c2585a2af" candidate_uuid: "77bab1bee7354d2fab28e2d8df23c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508074b997654a80863c999608271c26" is_pre_election: true
I20260925 14:55:15.424230 12106 raft_consensus.cc:2781] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:15.424237 11804 raft_consensus.cc:2454] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bab1bee7354d2fab28e2d8df23c472 in term 0.
I20260925 14:55:15.424260 12106 raft_consensus.cc:482] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:15.424283 12106 raft_consensus.cc:3037] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:15.425006 12106 raft_consensus.cc:504] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.425212 12106 leader_election.cc:288] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [CANDIDATE]: Term 1 election: Requested vote from peers 317b3172f04f43629f126edce41068da (127.9.141.65:37607), 508074b997654a80863c999608271c26 (127.9.141.66:42163)
I20260925 14:55:15.425305 11664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195a89e8ddf4d09af52382c2585a2af" candidate_uuid: "77bab1bee7354d2fab28e2d8df23c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "317b3172f04f43629f126edce41068da"
I20260925 14:55:15.425383 11664 raft_consensus.cc:3037] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:15.425998 11664 raft_consensus.cc:2454] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bab1bee7354d2fab28e2d8df23c472 in term 1.
I20260925 14:55:15.426113 11804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195a89e8ddf4d09af52382c2585a2af" candidate_uuid: "77bab1bee7354d2fab28e2d8df23c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508074b997654a80863c999608271c26"
I20260925 14:55:15.426174 11804 raft_consensus.cc:3037] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:15.426188 11875 leader_election.cc:302] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317b3172f04f43629f126edce41068da, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:15.426299 12106 raft_consensus.cc:2781] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:15.426437 12106 raft_consensus.cc:686] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 1 LEADER]: Becoming Leader. State: Replica: 77bab1bee7354d2fab28e2d8df23c472, State: Running, Role: LEADER
I20260925 14:55:15.426508 12106 consensus_queue.cc:237] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.427037 11804 raft_consensus.cc:2454] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bab1bee7354d2fab28e2d8df23c472 in term 1.
I20260925 14:55:15.427258 11507 catalog_manager.cc:5495] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "77bab1bee7354d2fab28e2d8df23c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } health_report { overall_health: HEALTHY } } }
I20260925 14:55:15.436643 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.434049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.9.141.67:43743)
0925 14:55:15.434186 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.434188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.434195 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:15.434224 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.434225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.434241 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.434312 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:15.434435 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.434944 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.435749 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.435963 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.436026 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.436165 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.436268 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.436327 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.436446 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.436466 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
I20260925 14:55:15.441401 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.439021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:48419 (local address 127.9.141.65:37607)
0925 14:55:15.439079 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.439081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.439183 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:55:15.439208 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.439208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.439218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.439270 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:15.439411 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.439878 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.440772 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.440976 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.441037 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.441209 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.441233 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.441273 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.441336 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.441352 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:55:15.441459 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.438928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.9.141.67:48419)
0925 14:55:15.439084 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:15.439095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:15.439158 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:15.439278 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:15.439279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:15.439280 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:15.439327 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.439327 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.439951 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.439951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.440667 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.440669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.441068 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.441069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.441121 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.441133 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.441288 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.441290 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:15.441333 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:55:15.441365 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260925 14:55:15.443388 11664 raft_consensus.cc:1261] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Refusing update from remote peer 77bab1bee7354d2fab28e2d8df23c472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:15.443392 11804 raft_consensus.cc:1261] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Refusing update from remote peer 77bab1bee7354d2fab28e2d8df23c472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:15.443666 12111 consensus_queue.cc:1035] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:15.443819 12106 consensus_queue.cc:1035] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:15.450929 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.448339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:45259 (local address 127.9.141.67:43743)
0925 14:55:15.448399 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.448402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.448567 (+ 165us) server_negotiation.cc:407] Connection header received
0925 14:55:15.448646 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.448647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.448655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.448695 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:15.448780 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.449327 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.450270 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.450613 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.450663 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.450777 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.450798 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.450830 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.450872 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.450885 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:15.450958 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.448294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:43743 (local address 127.9.141.65:45259)
0925 14:55:15.448487 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:15.448498 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:15.448581 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:15.448696 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:15.448697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:15.448698 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:15.448734 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.448736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.449408 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.449409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.450189 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.450191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.450669 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.450670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.450715 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.450720 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.450831 (+ 111us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.450832 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:55:15.450869 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 14:55:15.450901 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260925 14:55:15.457458 11507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:15.457624 11507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:15.458256 11507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Sending DeleteTablet for 3 replicas of tablet 4195a89e8ddf4d09af52382c2585a2af
I20260925 14:55:15.458498 11784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195a89e8ddf4d09af52382c2585a2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:15 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:56828
I20260925 14:55:15.458536 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195a89e8ddf4d09af52382c2585a2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:15 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55424
I20260925 14:55:15.458606 12089 tablet_replica.cc:330] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: stopping tablet replica
I20260925 14:55:15.458539 11643 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195a89e8ddf4d09af52382c2585a2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:15 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:44984
I20260925 14:55:15.458655 12089 raft_consensus.cc:2229] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:15.458688 12089 raft_consensus.cc:2258] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:15.458714 12090 tablet_replica.cc:330] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: stopping tablet replica
I20260925 14:55:15.458715 12091 tablet_replica.cc:330] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: stopping tablet replica
I20260925 14:55:15.458776 12090 raft_consensus.cc:2229] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:15.458778 12091 raft_consensus.cc:2229] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:15.458817 12090 raft_consensus.cc:2258] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:15.458860 12091 raft_consensus.cc:2258] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:15.459012 12089 ts_tablet_manager.cc:1905] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:15.459282 12091 ts_tablet_manager.cc:1905] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:15.459345 11507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:15.459484 11507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:15.460927 12090 ts_tablet_manager.cc:1905] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:15.461470 12089 ts_tablet_manager.cc:1918] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:15.461524 12089 log.cc:1196] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/4195a89e8ddf4d09af52382c2585a2af
I20260925 14:55:15.461753 12089 ts_tablet_manager.cc:1939] T 4195a89e8ddf4d09af52382c2585a2af P 508074b997654a80863c999608271c26: Deleting consensus metadata
I20260925 14:55:15.461804 11924 tablet_service.cc:1467] Processing CreateTablet for tablet ea5de0aac8384707a27134013c2e2c2d (DEFAULT_TABLE table=test-table [id=13549e3a6a3a43f28be40cee4c32591e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:15.461918 11643 tablet_service.cc:1467] Processing CreateTablet for tablet ea5de0aac8384707a27134013c2e2c2d (DEFAULT_TABLE table=test-table [id=13549e3a6a3a43f28be40cee4c32591e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:15.462029 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5de0aac8384707a27134013c2e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:15.462477 12090 ts_tablet_manager.cc:1918] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:15.462518 12090 log.cc:1196] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/4195a89e8ddf4d09af52382c2585a2af
I20260925 14:55:15.462713 12090 ts_tablet_manager.cc:1939] T 4195a89e8ddf4d09af52382c2585a2af P 317b3172f04f43629f126edce41068da: Deleting consensus metadata
I20260925 14:55:15.462929 12091 ts_tablet_manager.cc:1918] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:15.462984 12091 log.cc:1196] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/4195a89e8ddf4d09af52382c2585a2af
I20260925 14:55:15.463182 12091 ts_tablet_manager.cc:1939] T 4195a89e8ddf4d09af52382c2585a2af P 77bab1bee7354d2fab28e2d8df23c472: Deleting consensus metadata
I20260925 14:55:15.463295 12101 tablet_bootstrap.cc:492] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da: Bootstrap starting.
I20260925 14:55:15.463335 11924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5de0aac8384707a27134013c2e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:15.463553 11784 tablet_service.cc:1467] Processing CreateTablet for tablet ea5de0aac8384707a27134013c2e2c2d (DEFAULT_TABLE table=test-table [id=13549e3a6a3a43f28be40cee4c32591e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:15.463601 12101 tablet_bootstrap.cc:654] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:15.463650 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea5de0aac8384707a27134013c2e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:15.464120 12101 tablet_bootstrap.cc:492] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da: No bootstrap required, opened a new log
I20260925 14:55:15.464170 12101 ts_tablet_manager.cc:1397] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:15.464304 12101 raft_consensus.cc:348] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.464366 12101 raft_consensus.cc:374] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:15.464419 12101 raft_consensus.cc:729] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317b3172f04f43629f126edce41068da, State: Initialized, Role: FOLLOWER
I20260925 14:55:15.464442 11493 catalog_manager.cc:4841] TS 508074b997654a80863c999608271c26 (127.9.141.66:42163): tablet 4195a89e8ddf4d09af52382c2585a2af (table test-table [id=93b453cfb91b4d8889a832c12aa5fadb]) successfully deleted
I20260925 14:55:15.464464 12101 consensus_queue.cc:260] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.464578 12101 ts_tablet_manager.cc:1428] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:15.464702 12100 tablet_bootstrap.cc:492] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Bootstrap starting.
I20260925 14:55:15.465018 11491 catalog_manager.cc:4841] TS 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743): tablet 4195a89e8ddf4d09af52382c2585a2af (table test-table [id=93b453cfb91b4d8889a832c12aa5fadb]) successfully deleted
I20260925 14:55:15.465076 12100 tablet_bootstrap.cc:654] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:15.465296 11491 catalog_manager.cc:4841] TS 317b3172f04f43629f126edce41068da (127.9.141.65:37607): tablet 4195a89e8ddf4d09af52382c2585a2af (table test-table [id=93b453cfb91b4d8889a832c12aa5fadb]) successfully deleted
I20260925 14:55:15.465461 12102 tablet_bootstrap.cc:492] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Bootstrap starting.
I20260925 14:55:15.465772 12102 tablet_bootstrap.cc:654] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:15.466284 12102 tablet_bootstrap.cc:492] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: No bootstrap required, opened a new log
I20260925 14:55:15.466351 12102 ts_tablet_manager.cc:1397] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:15.466504 12102 raft_consensus.cc:348] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.466589 12102 raft_consensus.cc:374] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:15.466626 12102 raft_consensus.cc:729] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 508074b997654a80863c999608271c26, State: Initialized, Role: FOLLOWER
I20260925 14:55:15.466679 12102 consensus_queue.cc:260] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.466780 12102 ts_tablet_manager.cc:1428] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:15.468849 12100 tablet_bootstrap.cc:492] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: No bootstrap required, opened a new log
I20260925 14:55:15.468899 12100 ts_tablet_manager.cc:1397] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 14:55:15.469115 12100 raft_consensus.cc:348] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.469178 12100 raft_consensus.cc:374] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:15.469193 12100 raft_consensus.cc:729] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bab1bee7354d2fab28e2d8df23c472, State: Initialized, Role: FOLLOWER
I20260925 14:55:15.469249 12100 consensus_queue.cc:260] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.469412 12100 ts_tablet_manager.cc:1428] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:15.551060 12076 raft_consensus.cc:482] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:15.551167 12076 raft_consensus.cc:504] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.551347 12076 leader_election.cc:288] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 508074b997654a80863c999608271c26 (127.9.141.66:42163), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:15.551573 11804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5de0aac8384707a27134013c2e2c2d" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508074b997654a80863c999608271c26" is_pre_election: true
I20260925 14:55:15.551678 11804 raft_consensus.cc:2454] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 0.
I20260925 14:55:15.551702 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5de0aac8384707a27134013c2e2c2d" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472" is_pre_election: true
I20260925 14:55:15.551784 11944 raft_consensus.cc:2454] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 0.
I20260925 14:55:15.551848 11597 leader_election.cc:302] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317b3172f04f43629f126edce41068da, 508074b997654a80863c999608271c26; no voters:
I20260925 14:55:15.551996 12076 raft_consensus.cc:2781] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:15.552044 12076 raft_consensus.cc:482] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:15.552070 12076 raft_consensus.cc:3037] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:15.552961 12076 raft_consensus.cc:504] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.553129 12076 leader_election.cc:288] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 election: Requested vote from peers 508074b997654a80863c999608271c26 (127.9.141.66:42163), 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743)
I20260925 14:55:15.553294 11804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5de0aac8384707a27134013c2e2c2d" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "508074b997654a80863c999608271c26"
I20260925 14:55:15.553318 11944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea5de0aac8384707a27134013c2e2c2d" candidate_uuid: "317b3172f04f43629f126edce41068da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bab1bee7354d2fab28e2d8df23c472"
I20260925 14:55:15.553381 11804 raft_consensus.cc:3037] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:15.553385 11944 raft_consensus.cc:3037] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:15.554198 11944 raft_consensus.cc:2454] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 1.
I20260925 14:55:15.554271 11804 raft_consensus.cc:2454] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 317b3172f04f43629f126edce41068da in term 1.
I20260925 14:55:15.554431 11595 leader_election.cc:302] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 317b3172f04f43629f126edce41068da, 77bab1bee7354d2fab28e2d8df23c472; no voters:
I20260925 14:55:15.554554 12076 raft_consensus.cc:2781] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:15.554605 12076 raft_consensus.cc:686] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [term 1 LEADER]: Becoming Leader. State: Replica: 317b3172f04f43629f126edce41068da, State: Running, Role: LEADER
I20260925 14:55:15.554648 12076 consensus_queue.cc:237] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } }
I20260925 14:55:15.555217 11507 catalog_manager.cc:5495] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da reported cstate change: term changed from 0 to 1, leader changed from <none> to 317b3172f04f43629f126edce41068da (127.9.141.65). New cstate: current_term: 1 leader_uuid: "317b3172f04f43629f126edce41068da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "317b3172f04f43629f126edce41068da" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 37607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:15.584699 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.582387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.9.141.65:37607)
0925 14:55:15.582429 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.582431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.582439 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:15.582464 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.582465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.582475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.582608 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:15.582717 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.583200 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.583958 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.584170 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.584248 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.584410 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.584443 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.584529 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.584595 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.584616 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:15.588874 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.586762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:54339 (local address 127.9.141.65:37607)
0925 14:55:15.586791 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:15.586793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:15.586812 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:55:15.586828 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:15.586828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:15.586835 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:15.586870 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:15.587024 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.587472 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.588196 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.588362 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.588464 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.588605 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.588641 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.588704 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:15.588784 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:55:15.588800 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 14:55:15.588881 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:15.586657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.9.141.65:54339)
0925 14:55:15.586719 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:15.586730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:15.586768 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:15.586889 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:15.586889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:15.586890 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:15.586945 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.586946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.587606 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.587606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.588134 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:15.588136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:15.588464 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:15.588465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:15.588511 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:15.588517 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:15.588706 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:15.588708 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:15.588780 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:55:15.588820 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:55:15.590751 11944 raft_consensus.cc:1261] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Refusing update from remote peer 317b3172f04f43629f126edce41068da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:15.590821 11804 raft_consensus.cc:1261] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Refusing update from remote peer 317b3172f04f43629f126edce41068da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:15.591028 12036 consensus_queue.cc:1035] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bab1bee7354d2fab28e2d8df23c472" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:15.591140 12076 consensus_queue.cc:1035] T ea5de0aac8384707a27134013c2e2c2d P 317b3172f04f43629f126edce41068da [LEADER]: Connected to new peer: Peer: permanent_uuid: "508074b997654a80863c999608271c26" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:55:15.595323 12069 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:55:17.121605 11507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:17.121717 11507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:17.122597 11507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04854b541b2d4ed2acb76311fd752337: Sending DeleteTablet for 3 replicas of tablet ea5de0aac8384707a27134013c2e2c2d
I20260925 14:55:17.122869 11784 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5de0aac8384707a27134013c2e2c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:17 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:56828
I20260925 14:55:17.122915 11644 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5de0aac8384707a27134013c2e2c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:17 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:44984
I20260925 14:55:17.122903 11924 tablet_service.cc:1514] Processing DeleteTablet for tablet ea5de0aac8384707a27134013c2e2c2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:17 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55424
I20260925 14:55:17.123119 12153 tablet_replica.cc:330] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: stopping tablet replica
I20260925 14:55:17.123188 12153 raft_consensus.cc:2229] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:17.123203 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 11579
I20260925 14:55:17.123225 12153 raft_consensus.cc:2258] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:17.124675 12151 tablet_replica.cc:330] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: stopping tablet replica
I20260925 14:55:17.124761 12151 raft_consensus.cc:2229] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:17.124840 12151 raft_consensus.cc:2258] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:17.125080 12151 ts_tablet_manager.cc:1905] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:17.126878 12151 ts_tablet_manager.cc:1918] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:55:17.127065 12151 log.cc:1196] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/ea5de0aac8384707a27134013c2e2c2d
I20260925 14:55:17.127267 12151 ts_tablet_manager.cc:1939] T ea5de0aac8384707a27134013c2e2c2d P 508074b997654a80863c999608271c26: Deleting consensus metadata
I20260925 14:55:17.127614 11493 catalog_manager.cc:4841] TS 508074b997654a80863c999608271c26 (127.9.141.66:42163): tablet ea5de0aac8384707a27134013c2e2c2d (table test-table [id=13549e3a6a3a43f28be40cee4c32591e]) successfully deleted
I20260925 14:55:17.128671 12153 ts_tablet_manager.cc:1905] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:55:17.129952 11491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:37607 (error 108)
I20260925 14:55:17.130260 12153 ts_tablet_manager.cc:1918] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:55:17.130283 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 11719
I20260925 14:55:17.130334 12153 log.cc:1196] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/ea5de0aac8384707a27134013c2e2c2d
I20260925 14:55:17.130554 12153 ts_tablet_manager.cc:1939] T ea5de0aac8384707a27134013c2e2c2d P 77bab1bee7354d2fab28e2d8df23c472: Deleting consensus metadata
I20260925 14:55:17.136696 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 11859
I20260925 14:55:17.136698 11491 catalog_manager.cc:4841] TS 77bab1bee7354d2fab28e2d8df23c472 (127.9.141.67:43743): tablet ea5de0aac8384707a27134013c2e2c2d (table test-table [id=13549e3a6a3a43f28be40cee4c32591e]) successfully deleted
I20260925 14:55:17.136773 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:17.136491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:37607 (local address 127.0.0.1:34862)
0925 14:55:17.136666 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:17.136727 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:37607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
W20260925 14:55:17.138358 11491 catalog_manager.cc:4584] TS 317b3172f04f43629f126edce41068da (127.9.141.65:37607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea5de0aac8384707a27134013c2e2c2d: Network error: Client connection negotiation failed: client connection to 127.9.141.65:37607: connect: Connection refused (error 111)
I20260925 14:55:17.142158 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 11475
2026-09-25T14:55:17Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5522 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:17 dist-test-slave-1sv7 krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:17 dist-test-slave-1sv7 krb5kdc[12158](info): set up 2 sockets
Sep 25 14:55:17 dist-test-slave-1sv7 krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260925 14:55:18.184126 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:18.307549 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39727
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:18.387007 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:18.387243 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:18.387282 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:18.388432 12171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:18.388495 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:18.388520 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:18.388546 12171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:18.388578 12171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:18.390027 12171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:39727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:18.390291 12171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:18.390512 12171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:55:18.395289 12171 server_base.cc:1034] running on GCE node
W20260925 14:55:18.395303 12179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.395313 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.395437 12177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:18.395737 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:18.396016 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:18.397158 12171 hybrid_clock.cc:648] HybridClock initialized: now 1790348118397146 us; error 78 us; skew 500 ppm
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:18.399655 12171 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:18.400075 12171 webserver.cc:460] Webserver started at http://127.9.141.126:35731/ using document root <none> and password file <none>
I20260925 14:55:18.400283 12171 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:18.400329 12171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:18.400691 12171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:18.402004 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "45ce9c988549495081b33cb9b4ddf1a1"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "e6fbd8b427f2492d62191bf3645a9ac6"
server_key_iv: "47cd65ee4eeef1b81a0b8016311761ef"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.402350 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "45ce9c988549495081b33cb9b4ddf1a1"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "e6fbd8b427f2492d62191bf3645a9ac6"
server_key_iv: "47cd65ee4eeef1b81a0b8016311761ef"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.403590 12171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:18.404260 12186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.404479 12171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:18.404548 12171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "45ce9c988549495081b33cb9b4ddf1a1"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "e6fbd8b427f2492d62191bf3645a9ac6"
server_key_iv: "47cd65ee4eeef1b81a0b8016311761ef"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.404651 12171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:18.430316 12171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:18.432799 12171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:18.432953 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:18.436815 12171 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:39727
I20260925 14:55:18.436854 12238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:39727 every 8 connection(s)
I20260925 14:55:18.437378 12171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:18.438019 12239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:18.440275 12239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1: Bootstrap starting.
I20260925 14:55:18.440888 12239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:18.441176 12239 log.cc:824] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:18.441890 12239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1: No bootstrap required, opened a new log
I20260925 14:55:18.442565 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12171
I20260925 14:55:18.442679 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:18.442865 9781 external_mini_cluster.cc:1428] Setting key ccd1f29e0dd86307483331d94e70b0ec
I20260925 14:55:18.443924 12239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ce9c988549495081b33cb9b4ddf1a1" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39727 } }
I20260925 14:55:18.444088 12239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:18.444126 12239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ce9c988549495081b33cb9b4ddf1a1, State: Initialized, Role: FOLLOWER
I20260925 14:55:18.444293 12239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ce9c988549495081b33cb9b4ddf1a1" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39727 } }
I20260925 14:55:18.444418 12239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:18.444464 12239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:18.444519 12239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:18.445566 12239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ce9c988549495081b33cb9b4ddf1a1" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39727 } }
I20260925 14:55:18.445652 12239 leader_election.cc:302] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ce9c988549495081b33cb9b4ddf1a1; no voters:
I20260925 14:55:18.445825 12239 leader_election.cc:288] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:18.445858 12244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:18.445958 12244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [term 1 LEADER]: Becoming Leader. State: Replica: 45ce9c988549495081b33cb9b4ddf1a1, State: Running, Role: LEADER
I20260925 14:55:18.446038 12244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ce9c988549495081b33cb9b4ddf1a1" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39727 } }
I20260925 14:55:18.446081 12239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:18.447274 12245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ce9c988549495081b33cb9b4ddf1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ce9c988549495081b33cb9b4ddf1a1" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39727 } } }
I20260925 14:55:18.447381 12245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:18.447625 12246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ce9c988549495081b33cb9b4ddf1a1. Latest consensus state: current_term: 1 leader_uuid: "45ce9c988549495081b33cb9b4ddf1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ce9c988549495081b33cb9b4ddf1a1" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39727 } } }
I20260925 14:55:18.447934 12246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1 [sys.catalog]: This master's current role is: LEADER
W20260925 14:55:18.447973 12258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:18.448009 12258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:18.448215 12259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:18.448441 12259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:18.450352 12259 catalog_manager.cc:1347] Generated new cluster ID: 2921a4fb108d447e836b4a30b3d5fe04
I20260925 14:55:18.450398 12259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:18.454442 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.443380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.9.141.126:39727)
0925 14:55:18.443572 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:18.443580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:18.443603 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:55:18.443913 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:18.443920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:18.443971 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:18.444106 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:18.444930 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.446009 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.446893 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.447537 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.447646 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.450722 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:18.450726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:18.450726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:18.450736 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:18.452641 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.453005 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.453007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.453008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.453124 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.453300 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.453300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.453301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.453571 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:18.453704 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:18.454094 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 14:55:18.454253 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260925 14:55:18.457372 12259 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:18.458076 12259 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:18.464066 12259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ce9c988549495081b33cb9b4ddf1a1: Generated new TSK 0
I20260925 14:55:18.464284 12259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:18.496528 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39727
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:18.577117 12265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:18.577318 12265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:18.577340 12265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:18.578542 12265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:18.578585 12265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:18.578611 12265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:18.579958 12265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:18.580138 12265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:18.580312 12265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:18.585880 12271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.585887 12270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.585932 12273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:18.585933 12265 server_base.cc:1034] running on GCE node
I20260925 14:55:18.586441 12265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:18.586706 12265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:18.587888 12265 hybrid_clock.cc:648] HybridClock initialized: now 1790348118587890 us; error 43 us; skew 500 ppm
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:18.590493 12265 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:18.590976 12265 webserver.cc:460] Webserver started at http://127.9.141.65:35887/ using document root <none> and password file <none>
I20260925 14:55:18.591207 12265 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:18.591256 12265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:18.591387 12265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:18.592618 12265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "8364416d0bff4d97bf75084cfeef3883"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "9d53adc170a068cad7506b940bf18a84"
server_key_iv: "8061272f09066754cf068d6132ce9546"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.592994 12265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "8364416d0bff4d97bf75084cfeef3883"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "9d53adc170a068cad7506b940bf18a84"
server_key_iv: "8061272f09066754cf068d6132ce9546"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.594357 12265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:18.595244 12280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.595453 12265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:18.595559 12265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "8364416d0bff4d97bf75084cfeef3883"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "9d53adc170a068cad7506b940bf18a84"
server_key_iv: "8061272f09066754cf068d6132ce9546"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.595633 12265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:18.613739 12265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:18.616473 12265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:18.616662 12265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:18.616937 12265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:18.617291 12265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:18.617347 12265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.617383 12265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:18.617410 12265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.623131 12265 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:43453
I20260925 14:55:18.623255 12393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:43453 every 8 connection(s)
I20260925 14:55:18.623515 12265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:18.631543 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12265
I20260925 14:55:18.631644 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:18.631863 9781 external_mini_cluster.cc:1428] Setting key b77987eb5a8a42e0fd7a41be21dba0ae
I20260925 14:55:18.633100 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.624316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:53837 (local address 127.9.141.126:39727)
0925 14:55:18.624410 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:18.624412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:18.624570 (+ 158us) server_negotiation.cc:407] Connection header received
0925 14:55:18.625094 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:18.625095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:18.625107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:18.625164 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:18.626163 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.626674 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.627679 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.627914 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.627993 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.630083 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:18.630084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:18.630084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:18.630091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:18.631428 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.631960 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.631961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.631962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.632049 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.632395 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.632396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.632396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.632533 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:18.632607 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:18.632964 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 14:55:18.633035 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:55:18.633380 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.624246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39727 (local address 127.9.141.65:53837)
0925 14:55:18.624442 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:18.624486 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:18.624988 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:18.625241 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:18.625245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:18.625862 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:18.626061 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:18.626066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.626772 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.626774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:18.627584 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:18.627586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.627999 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.628000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:18.628135 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.628646 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:18.628654 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:18.629994 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:18.631523 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:18.631531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:18.631533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:18.631879 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:18.632151 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:18.632152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:18.632152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:18.632306 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:18.632613 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:18.632620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:18.632820 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 14:55:18.632941 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 14:55:18.633127 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260925 14:55:18.633909 12394 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39727
I20260925 14:55:18.634017 12394 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:18.634215 12394 heartbeater.cc:511] Master 127.9.141.126:39727 requested a full tablet report, sending...
I20260925 14:55:18.634810 12203 ts_manager.cc:194] Registered new tserver with Master: 8364416d0bff4d97bf75084cfeef3883 (127.9.141.65:43453)
I20260925 14:55:18.635712 12203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:53837
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:18.659475 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39727
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:18.740253 12399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:18.740527 12399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:18.740566 12399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:18.741736 12399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:18.741803 12399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:18.741844 12399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:18.743283 12399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:18.743546 12399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:18.743769 12399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:18.748713 12405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.748700 12404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:18.748906 12399 server_base.cc:1034] running on GCE node
W20260925 14:55:18.748713 12407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:18.749166 12399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:18.749405 12399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:18.750566 12399 hybrid_clock.cc:648] HybridClock initialized: now 1790348118750554 us; error 28 us; skew 500 ppm
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:18.753058 12399 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:18.753551 12399 webserver.cc:460] Webserver started at http://127.9.141.66:40161/ using document root <none> and password file <none>
I20260925 14:55:18.753852 12399 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:18.753935 12399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:18.754072 12399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:18.755275 12399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "5f13c693318a400f8e723923012871c4"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "573c7a3383f133a20455c16b1e636272"
server_key_iv: "17e178c55e9824c9a44a5bb685c76939"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.755635 12399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "5f13c693318a400f8e723923012871c4"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "573c7a3383f133a20455c16b1e636272"
server_key_iv: "17e178c55e9824c9a44a5bb685c76939"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.757009 12399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:55:18.757700 12414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.757872 12399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:18.757969 12399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "5f13c693318a400f8e723923012871c4"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "573c7a3383f133a20455c16b1e636272"
server_key_iv: "17e178c55e9824c9a44a5bb685c76939"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.758052 12399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:18.781337 12399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:18.783609 12399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:18.783778 12399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:18.784024 12399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:18.784328 12399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:18.784380 12399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.784444 12399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:18.784475 12399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.789942 12399 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:45901
I20260925 14:55:18.790036 12527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:45901 every 8 connection(s)
I20260925 14:55:18.790328 12399 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:18.794183 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12399
I20260925 14:55:18.794277 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:18.794461 9781 external_mini_cluster.cc:1428] Setting key 7d165019a9db19882e7feb4134494858
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:18.801378 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.791443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:54201 (local address 127.9.141.126:39727)
0925 14:55:18.791517 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:18.791519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:18.791536 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:55:18.792169 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:18.792170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:18.792182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:18.792264 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:18.793562 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.794370 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.795606 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.795921 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.795997 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.798385 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:18.798387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:18.798388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:18.798395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:18.799889 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.800311 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.800313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.800313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.800373 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.800671 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.800672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.800672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.800790 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:18.800862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:18.801142 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:55:18.801285 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:18.801512 12530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.791090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39727 (local address 127.9.141.66:54201)
0925 14:55:18.791298 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:18.791370 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:18.792048 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:18.792354 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:18.792358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:18.793190 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:18.793453 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:18.793458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.794465 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.794468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:18.795522 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:18.795524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.796009 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.796009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:18.796107 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.796748 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:18.796756 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:18.798250 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:18.799983 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:18.799988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:18.799991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:18.800219 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:18.800465 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:18.800466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:18.800466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:18.800579 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:18.800864 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:18.800870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:18.801013 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:55:18.801121 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:55:18.801286 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
I20260925 14:55:18.801955 12528 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39727
I20260925 14:55:18.802047 12528 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:18.802244 12528 heartbeater.cc:511] Master 127.9.141.126:39727 requested a full tablet report, sending...
I20260925 14:55:18.802637 12203 ts_manager.cc:194] Registered new tserver with Master: 5f13c693318a400f8e723923012871c4 (127.9.141.66:45901)
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:55:18.803179 12203 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:54201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:18.822961 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39727
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:18.903674 12533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:18.903900 12533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:18.903923 12533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:18.905099 12533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:18.905140 12533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:18.905192 12533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:18.906683 12533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:18.906901 12533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:18.907094 12533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:18.911911 12538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.911973 12539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:18.911989 12541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:18.912434 12533 server_base.cc:1034] running on GCE node
I20260925 14:55:18.912618 12533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:18.912842 12533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:18.913985 12533 hybrid_clock.cc:648] HybridClock initialized: now 1790348118913957 us; error 44 us; skew 500 ppm
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:18.916635 12533 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:18.917073 12533 webserver.cc:460] Webserver started at http://127.9.141.67:33435/ using document root <none> and password file <none>
I20260925 14:55:18.917307 12533 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:18.917352 12533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:18.917500 12533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:18.918604 12533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "5786d68666b44409ba0e670ef8966198"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "e68044c0b3b4a1421829234d094f4430"
server_key_iv: "faa88ec38f0e6dac130e9530eaee9b43"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.918967 12533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "5786d68666b44409ba0e670ef8966198"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "e68044c0b3b4a1421829234d094f4430"
server_key_iv: "faa88ec38f0e6dac130e9530eaee9b43"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.920534 12533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 14:55:18.921406 12548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.921639 12533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:18.921743 12533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "5786d68666b44409ba0e670ef8966198"
format_stamp: "Formatted at 2026-09-25 14:55:18 on dist-test-slave-1sv7"
server_key: "e68044c0b3b4a1421829234d094f4430"
server_key_iv: "faa88ec38f0e6dac130e9530eaee9b43"
server_key_version: "encryptionkey@0"
I20260925 14:55:18.921831 12533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:18.929421 12533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:18.931456 12533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:18.931617 12533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:18.931844 12533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:18.932152 12533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:18.932204 12533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.932255 12533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:18.932279 12533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:18.938194 12533 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:35393
I20260925 14:55:18.938246 12661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:35393 every 8 connection(s)
I20260925 14:55:18.938580 12533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:55:18 dist-test-slave-1sv7 krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348118, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:18.948344 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12533
I20260925 14:55:18.948468 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:18.948542 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.939279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:60577 (local address 127.9.141.126:39727)
0925 14:55:18.939367 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:18.939369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:18.939560 (+ 191us) server_negotiation.cc:407] Connection header received
0925 14:55:18.940086 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:18.940087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:18.940099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:18.940156 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:18.941170 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.941676 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.942674 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.942886 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.942977 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.945472 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:18.945474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:18.945474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:18.945482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:18.947121 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.947491 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.947492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.947492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.947544 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:18.947871 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:18.947872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:18.947872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:18.947981 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:18.948036 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:18.948323 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:55:18.948429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:55:18.948675 9781 external_mini_cluster.cc:1428] Setting key ccaa6eea999e8b683203096723656e1a
I20260925 14:55:18.948722 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:18.939271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39727 (local address 127.9.141.67:60577)
0925 14:55:18.939420 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:18.939474 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:18.939979 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:18.940229 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:18.940233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:18.940867 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:18.941055 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:18.941059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.941774 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.941775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:18.942570 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:18.942572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:18.942987 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:18.942988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:18.943125 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:18.943846 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:18.943855 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:18.945294 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:18.947194 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:18.947200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:18.947202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:18.947421 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:18.947638 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:18.947639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:18.947639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:18.947794 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:18.948041 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:18.948046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:18.948163 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 14:55:18.948244 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:55:18.948414 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260925 14:55:18.949254 12662 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39727
I20260925 14:55:18.949360 12662 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:18.949532 12662 heartbeater.cc:511] Master 127.9.141.126:39727 requested a full tablet report, sending...
I20260925 14:55:18.949884 12202 ts_manager.cc:194] Registered new tserver with Master: 5786d68666b44409ba0e670ef8966198 (127.9.141.67:35393)
I20260925 14:55:18.950217 12202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:60577
I20260925 14:55:18.960783 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:55:18.972114 12243 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:18.971116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.9.141.126:39727)
0925 14:55:18.971215 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:18.971217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:18.971325 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:55:18.971415 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:18.971415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:18.971433 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:18.971499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:18.972059 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43418: BlockingRecv error: recv got EOF from 127.0.0.1:43418 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:55:18.973052 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12265
I20260925 14:55:18.977200 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12399
I20260925 14:55:18.981428 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12533
I20260925 14:55:18.985432 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12171
2026-09-25T14:55:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1841 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:19 dist-test-slave-1sv7 krb5kdc[12674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:19 dist-test-slave-1sv7 krb5kdc[12674](info): set up 2 sockets
Sep 25 14:55:19 dist-test-slave-1sv7 krb5kdc[12674](info): commencing operation
krb5kdc: starting...
W20260925 14:55:20.016215 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:20 dist-test-slave-1sv7 krb5kdc[12674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:20.136011 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46345
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:46345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:20.214816 12687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:20.215055 12687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:20.215090 12687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:20.216243 12687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:20.216311 12687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:20.216336 12687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:20.216365 12687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:20.216388 12687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:20.217914 12687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:46345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:20.218197 12687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:20.218427 12687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:20.223163 12695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:20.223163 12692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:20.223205 12693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:20.223628 12687 server_base.cc:1034] running on GCE node
I20260925 14:55:20.223919 12687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:20.224149 12687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:20.225294 12687 hybrid_clock.cc:648] HybridClock initialized: now 1790348120225274 us; error 36 us; skew 500 ppm
Sep 25 14:55:20 dist-test-slave-1sv7 krb5kdc[12674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:20.227684 12687 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:20.228046 12687 webserver.cc:460] Webserver started at http://127.9.141.126:45907/ using document root <none> and password file <none>
I20260925 14:55:20.228214 12687 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:20.228257 12687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:20.228363 12687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:20.229452 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "3885d4db9cb047119055ac402d7ed837"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-1sv7"
server_key: "562916fd5528993a8a1b6e9d082ae63f"
server_key_iv: "afe40be90fc2cd2bf2ca83a3b23538ed"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.229763 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "3885d4db9cb047119055ac402d7ed837"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-1sv7"
server_key: "562916fd5528993a8a1b6e9d082ae63f"
server_key_iv: "afe40be90fc2cd2bf2ca83a3b23538ed"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.231084 12687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:55:20.231803 12702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:20.232023 12687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:20.232117 12687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "3885d4db9cb047119055ac402d7ed837"
format_stamp: "Formatted at 2026-09-25 14:55:20 on dist-test-slave-1sv7"
server_key: "562916fd5528993a8a1b6e9d082ae63f"
server_key_iv: "afe40be90fc2cd2bf2ca83a3b23538ed"
server_key_version: "encryptionkey@0"
I20260925 14:55:20.232192 12687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:20.250834 12687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:20.253139 12687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:20.253288 12687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:20.257185 12687 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:46345
I20260925 14:55:20.257310 12754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:46345 every 8 connection(s)
I20260925 14:55:20.257594 12687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:20.258486 12755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:20.260591 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12687
I20260925 14:55:20.260656 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:20.260887 9781 external_mini_cluster.cc:1428] Setting key 7c033cd77f02b310a03144b72200cc15
I20260925 14:55:20.261252 12755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837: Bootstrap starting.
I20260925 14:55:20.261982 12755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:20.262364 12755 log.cc:824] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:20.263201 12755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837: No bootstrap required, opened a new log
I20260925 14:55:20.265605 12755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3885d4db9cb047119055ac402d7ed837" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46345 } }
I20260925 14:55:20.265748 12755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:20.265770 12755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3885d4db9cb047119055ac402d7ed837, State: Initialized, Role: FOLLOWER
I20260925 14:55:20.265947 12755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3885d4db9cb047119055ac402d7ed837" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46345 } }
I20260925 14:55:20.266040 12755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:20.266065 12755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:20.266136 12755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:20.266772 12755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3885d4db9cb047119055ac402d7ed837" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46345 } }
I20260925 14:55:20.266872 12755 leader_election.cc:302] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3885d4db9cb047119055ac402d7ed837; no voters:
I20260925 14:55:20.267048 12755 leader_election.cc:288] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:20.267081 12760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:20.267195 12760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [term 1 LEADER]: Becoming Leader. State: Replica: 3885d4db9cb047119055ac402d7ed837, State: Running, Role: LEADER
I20260925 14:55:20.267302 12760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3885d4db9cb047119055ac402d7ed837" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46345 } }
I20260925 14:55:20.267338 12755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 14:55:20 dist-test-slave-1sv7 krb5kdc[12674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:20.268723 12761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3885d4db9cb047119055ac402d7ed837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3885d4db9cb047119055ac402d7ed837" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46345 } } }
I20260925 14:55:20.268831 12761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:20.268924 12762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3885d4db9cb047119055ac402d7ed837. Latest consensus state: current_term: 1 leader_uuid: "3885d4db9cb047119055ac402d7ed837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3885d4db9cb047119055ac402d7ed837" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46345 } } }
I20260925 14:55:20.268999 12762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837 [sys.catalog]: This master's current role is: LEADER
W20260925 14:55:20.269373 12774 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:20.269598 12774 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:20.269753 12775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:20.269960 12775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:20.271541 12775 catalog_manager.cc:1347] Generated new cluster ID: a1c3b4b019d94b91ae7b6e6f43f532a4
I20260925 14:55:20.271600 12775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:20.273002 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:20.261391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.9.141.126:46345)
0925 14:55:20.261567 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:20.261575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:20.261599 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:55:20.262049 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:20.262056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:20.262134 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:20.262272 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:20.263151 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.264065 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.265350 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.265795 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.265923 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:20.268568 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:20.268572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:20.268572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:20.268588 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:20.270668 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:20.271120 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:20.271123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:20.271124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:20.271232 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:20.271539 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:20.271540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:20.271540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:20.271899 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:20.272048 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:20.272619 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 14:55:20.272774 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260925 14:55:20.274009 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:55:20.297959 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:20.293533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.9.141.126:46345)
0925 14:55:20.293579 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:20.293581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:20.293670 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:55:20.293741 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:20.293742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:20.293754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:20.293816 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:20.294223 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.295116 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.296152 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:20.296579 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:20.296662 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:20.297428 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:20.297429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:20.297430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:20.297685 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:20.297706 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:20.297714 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:20.297784 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:20.297838 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:55:20.297866 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:20.306033 12775 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:20.306754 12775 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:20.328920 12775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3885d4db9cb047119055ac402d7ed837: Generated new TSK 0
I20260925 14:55:20.329149 12775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:20.340250 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12687
2026-09-25T14:55:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1353 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:20 dist-test-slave-1sv7 krb5kdc[12788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:20 dist-test-slave-1sv7 krb5kdc[12788](info): set up 2 sockets
Sep 25 14:55:20 dist-test-slave-1sv7 krb5kdc[12788](info): commencing operation
krb5kdc: starting...
W20260925 14:55:21.488404 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:21 dist-test-slave-1sv7 krb5kdc[12788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:21.612205 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36901
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:21.691854 12801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:21.692054 12801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:21.692075 12801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:21.693301 12801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:21.693347 12801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:21.693362 12801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:21.693374 12801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:21.693387 12801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:21.694867 12801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:21.695114 12801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:21.695346 12801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:21.700210 12806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:21.700210 12807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:21.700219 12809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:21.700464 12801 server_base.cc:1034] running on GCE node
I20260925 14:55:21.700698 12801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:21.700970 12801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:21.702173 12801 hybrid_clock.cc:648] HybridClock initialized: now 1790348121702147 us; error 40 us; skew 500 ppm
Sep 25 14:55:21 dist-test-slave-1sv7 krb5kdc[12788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:21.709324 12801 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:21.709794 12801 webserver.cc:460] Webserver started at http://127.9.141.126:32975/ using document root <none> and password file <none>
I20260925 14:55:21.710053 12801 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:21.710099 12801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:21.710228 12801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:21.711292 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "3187c2ceee9743f48ea87aec1b5c8b03"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-1sv7"
server_key: "5bc0dae7b5bf902c191f4e314d7b877e"
server_key_iv: "b4e3a225c6122c9a6deffe920181648d"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.711652 12801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "3187c2ceee9743f48ea87aec1b5c8b03"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-1sv7"
server_key: "5bc0dae7b5bf902c191f4e314d7b877e"
server_key_iv: "b4e3a225c6122c9a6deffe920181648d"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.713094 12801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:21.713913 12816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.714128 12801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:21.714226 12801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "3187c2ceee9743f48ea87aec1b5c8b03"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-1sv7"
server_key: "5bc0dae7b5bf902c191f4e314d7b877e"
server_key_iv: "b4e3a225c6122c9a6deffe920181648d"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.714306 12801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:21.744258 12801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:21.746290 12801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:21.746399 12801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:21.750375 12801 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:36901
I20260925 14:55:21.750428 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:36901 every 8 connection(s)
I20260925 14:55:21.750800 12801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:21.751746 12869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:21.754037 12869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03: Bootstrap starting.
I20260925 14:55:21.754612 12869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:21.754899 12869 log.cc:824] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:21.755677 12869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03: No bootstrap required, opened a new log
I20260925 14:55:21.757175 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12801
I20260925 14:55:21.757299 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:21.757472 9781 external_mini_cluster.cc:1428] Setting key 71eaf0cd9f95ba063335641b6751ad54
I20260925 14:55:21.757383 12869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36901 } }
I20260925 14:55:21.757521 12869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:21.757578 12869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3187c2ceee9743f48ea87aec1b5c8b03, State: Initialized, Role: FOLLOWER
I20260925 14:55:21.757725 12869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36901 } }
I20260925 14:55:21.757786 12869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:21.757828 12869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:21.757881 12869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:21.758733 12869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36901 } }
I20260925 14:55:21.758853 12869 leader_election.cc:302] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3187c2ceee9743f48ea87aec1b5c8b03; no voters:
I20260925 14:55:21.759053 12869 leader_election.cc:288] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:21.759078 12874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:21.759183 12874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [term 1 LEADER]: Becoming Leader. State: Replica: 3187c2ceee9743f48ea87aec1b5c8b03, State: Running, Role: LEADER
I20260925 14:55:21.759261 12874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36901 } }
I20260925 14:55:21.759307 12869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:21.759613 12875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3187c2ceee9743f48ea87aec1b5c8b03. Latest consensus state: current_term: 1 leader_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36901 } } }
I20260925 14:55:21.759706 12875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:21.759605 12876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3187c2ceee9743f48ea87aec1b5c8b03" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36901 } } }
I20260925 14:55:21.760118 12876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:21.760273 12880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:21.760790 12880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:21.762432 12880 catalog_manager.cc:1347] Generated new cluster ID: 688bc8c6919344bbbef221c032667ee9
I20260925 14:55:21.762480 12880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:21 dist-test-slave-1sv7 krb5kdc[12788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:21.768914 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.757991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.9.141.126:36901)
0925 14:55:21.758185 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.758195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.758225 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:55:21.758330 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.758336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.758376 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.758501 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.759306 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.760762 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.761749 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.762332 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.762439 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.765192 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.765196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.765196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.765207 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.767240 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.767627 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.767631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.767631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.767787 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.767998 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.767999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.767999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.768311 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.768501 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.768575 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.768713 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260925 14:55:21.772188 12880 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:21.772894 12880 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:21.785274 12880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3187c2ceee9743f48ea87aec1b5c8b03: Generated new TSK 0
I20260925 14:55:21.785550 12880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:21.816668 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36901
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:21.905517 12895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:21.905746 12895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:21.905782 12895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:21.906944 12895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:21.907015 12895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:21.907063 12895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:21.908602 12895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:21.908869 12895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:21.909106 12895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:21.914232 12901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:21.914248 12900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:21.914275 12903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:21.914465 12895 server_base.cc:1034] running on GCE node
I20260925 14:55:21.914763 12895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:21.915006 12895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:21.916170 12895 hybrid_clock.cc:648] HybridClock initialized: now 1790348121916152 us; error 28 us; skew 500 ppm
Sep 25 14:55:21 dist-test-slave-1sv7 krb5kdc[12788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:21.919104 12895 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:21.919574 12895 webserver.cc:460] Webserver started at http://127.9.141.65:33001/ using document root <none> and password file <none>
I20260925 14:55:21.919811 12895 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:21.919879 12895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:21.920024 12895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:21.921114 12895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "a1c86209c3bc4d9aba3f728064ecc422"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-1sv7"
server_key: "f015f52e08b848d3b214843d055a4d09"
server_key_iv: "9078d636cef8de97ad731d37413130a1"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.921458 12895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "a1c86209c3bc4d9aba3f728064ecc422"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-1sv7"
server_key: "f015f52e08b848d3b214843d055a4d09"
server_key_iv: "9078d636cef8de97ad731d37413130a1"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.922793 12895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:21.923547 12910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.923725 12895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:21.923821 12895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "a1c86209c3bc4d9aba3f728064ecc422"
format_stamp: "Formatted at 2026-09-25 14:55:21 on dist-test-slave-1sv7"
server_key: "f015f52e08b848d3b214843d055a4d09"
server_key_iv: "9078d636cef8de97ad731d37413130a1"
server_key_version: "encryptionkey@0"
I20260925 14:55:21.923905 12895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:21.937970 12895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:21.940227 12895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:21.940439 12895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:21.940694 12895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:21.940999 12895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:21.941056 12895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.941097 12895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:21.941126 12895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:21.946655 12895 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:35273
I20260925 14:55:21.946698 13023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:35273 every 8 connection(s)
I20260925 14:55:21.946988 12895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:21.951969 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 12895
I20260925 14:55:21.952131 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:21.952343 9781 external_mini_cluster.cc:1428] Setting key da3fdf04229262f9983eae172f706723
Sep 25 14:55:21 dist-test-slave-1sv7 krb5kdc[12788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348121, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:21.956660 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.947637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:36379 (local address 127.9.141.126:36901)
0925 14:55:21.947719 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:21.947722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:21.947934 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:55:21.948389 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:21.948390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:21.948404 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:21.948449 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:21.949350 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.949840 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.950803 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.950998 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.951055 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.953313 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:21.953314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:21.953314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:21.953322 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:21.954937 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.955447 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.955448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.955448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.955517 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:21.955895 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:21.955896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:21.955897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:21.956055 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:21.956129 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:21.956375 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:55:21.956508 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:55:21.956732 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:21.947613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36901 (local address 127.9.141.65:36379)
0925 14:55:21.947813 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:21.947847 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:21.948294 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:21.948514 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:21.948517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:21.949091 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:21.949270 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.949275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.949916 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.949917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.950722 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:21.950724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:21.951071 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:21.951072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:21.951164 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:21.951665 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:21.951673 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:21.953204 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:21.955024 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:21.955031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:21.955034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:21.955353 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:21.955665 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:21.955665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:21.955666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:21.955811 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:21.956129 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:21.956134 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:21.956243 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 14:55:21.956327 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:55:21.956512 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":132,"threads_started":1}
I20260925 14:55:21.957213 13024 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36901
I20260925 14:55:21.957324 13024 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:21.957504 13024 heartbeater.cc:511] Master 127.9.141.126:36901 requested a full tablet report, sending...
I20260925 14:55:21.958055 12833 ts_manager.cc:194] Registered new tserver with Master: a1c86209c3bc4d9aba3f728064ecc422 (127.9.141.65:35273)
I20260925 14:55:21.959133 12833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:36379
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:21.981115 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36901
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:22.063480 13029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:22.063688 13029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:22.063710 13029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:22.064796 13029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:22.064832 13029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:22.064858 13029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:22.066289 13029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:22.066464 13029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:22.066640 13029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:22.071821 13035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.071861 13037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.071842 13034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:22.072157 13029 server_base.cc:1034] running on GCE node
I20260925 14:55:22.072342 13029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:22.072593 13029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:22.073740 13029 hybrid_clock.cc:648] HybridClock initialized: now 1790348122073725 us; error 26 us; skew 500 ppm
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[12788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:22.076303 13029 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:22.076748 13029 webserver.cc:460] Webserver started at http://127.9.141.66:40651/ using document root <none> and password file <none>
I20260925 14:55:22.076979 13029 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:22.077046 13029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:22.077195 13029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:22.078225 13029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "b130a85d174748168ae2ae968595bd28"
format_stamp: "Formatted at 2026-09-25 14:55:22 on dist-test-slave-1sv7"
server_key: "a84c67c2cab796b0a0192d7d5b6302bf"
server_key_iv: "e2aaa3062d661f0e9b56e2ca16b4b2ad"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.078579 13029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "b130a85d174748168ae2ae968595bd28"
format_stamp: "Formatted at 2026-09-25 14:55:22 on dist-test-slave-1sv7"
server_key: "a84c67c2cab796b0a0192d7d5b6302bf"
server_key_iv: "e2aaa3062d661f0e9b56e2ca16b4b2ad"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.079942 13029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:22.080724 13044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.080884 13029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:22.080987 13029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "b130a85d174748168ae2ae968595bd28"
format_stamp: "Formatted at 2026-09-25 14:55:22 on dist-test-slave-1sv7"
server_key: "a84c67c2cab796b0a0192d7d5b6302bf"
server_key_iv: "e2aaa3062d661f0e9b56e2ca16b4b2ad"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.081059 13029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:22.095005 13029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:22.097324 13029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:22.097481 13029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:22.097709 13029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:22.098018 13029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:22.098070 13029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.098111 13029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:22.098140 13029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.103698 13029 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:34779
I20260925 14:55:22.103793 13157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:34779 every 8 connection(s)
I20260925 14:55:22.104108 13029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:22.105901 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13029
I20260925 14:55:22.105981 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:22.106168 9781 external_mini_cluster.cc:1428] Setting key 82664de8e09dbc9a8a33075771492895
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[12788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:22.114153 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:22.104933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:51185 (local address 127.9.141.126:36901)
0925 14:55:22.105009 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:22.105012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:22.105144 (+ 132us) server_negotiation.cc:407] Connection header received
0925 14:55:22.105654 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:22.105656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:22.105667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:22.105733 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:22.106837 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.107338 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.108328 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.108584 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.108671 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:22.111147 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:22.111148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:22.111148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:22.111155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:22.112649 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:22.113135 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:22.113136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:22.113137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:22.113216 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:22.113517 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:22.113518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:22.113518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:22.113627 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:22.113684 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:22.114017 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 14:55:22.114090 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:55:22.114434 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:22.104817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36901 (local address 127.9.141.66:51185)
0925 14:55:22.105012 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:22.105062 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:22.105551 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:22.105812 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:22.105816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:22.106545 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:22.106735 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:22.106740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.107432 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.107433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:22.108233 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:22.108235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.108681 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.108682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:22.108818 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:22.109357 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:22.109365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:22.111054 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:22.112727 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:22.112733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:22.112736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:22.113034 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:22.113297 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:22.113298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:22.113298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:22.113439 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:22.113701 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:22.113708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:22.113878 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:55:22.113998 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:55:22.114160 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
I20260925 14:55:22.115020 13158 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36901
I20260925 14:55:22.115103 13158 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:22.115324 13158 heartbeater.cc:511] Master 127.9.141.126:36901 requested a full tablet report, sending...
I20260925 14:55:22.115748 12833 ts_manager.cc:194] Registered new tserver with Master: b130a85d174748168ae2ae968595bd28 (127.9.141.66:34779)
I20260925 14:55:22.116170 12833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:51185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:22.131531 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36901
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:22.211985 13163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:22.212231 13163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:22.212281 13163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:22.213434 13163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:22.213505 13163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:22.213549 13163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:22.215018 13163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:22.215286 13163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:22.215507 13163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:22.220691 13168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:22.220691 13171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:22.220897 13163 server_base.cc:1034] running on GCE node
W20260925 14:55:22.220726 13169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:22.221169 13163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:22.221441 13163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:22.222605 13163 hybrid_clock.cc:648] HybridClock initialized: now 1790348122222601 us; error 37 us; skew 500 ppm
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[12788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:22.225189 13163 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:22.225637 13163 webserver.cc:460] Webserver started at http://127.9.141.67:41441/ using document root <none> and password file <none>
I20260925 14:55:22.225906 13163 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:22.225972 13163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:22.226112 13163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:22.227275 13163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "35ff3555ea144d8dae6737fcd8c46504"
format_stamp: "Formatted at 2026-09-25 14:55:22 on dist-test-slave-1sv7"
server_key: "545075aaae41d391ec91f02580e31987"
server_key_iv: "43bdf31333b915db1990fa463783ee98"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.227651 13163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "35ff3555ea144d8dae6737fcd8c46504"
format_stamp: "Formatted at 2026-09-25 14:55:22 on dist-test-slave-1sv7"
server_key: "545075aaae41d391ec91f02580e31987"
server_key_iv: "43bdf31333b915db1990fa463783ee98"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.229074 13163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:22.229918 13178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.230166 13163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:22.230283 13163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "35ff3555ea144d8dae6737fcd8c46504"
format_stamp: "Formatted at 2026-09-25 14:55:22 on dist-test-slave-1sv7"
server_key: "545075aaae41d391ec91f02580e31987"
server_key_iv: "43bdf31333b915db1990fa463783ee98"
server_key_version: "encryptionkey@0"
I20260925 14:55:22.230389 13163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:22.250804 13163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:22.253439 13163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:22.253623 13163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:22.253893 13163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:22.254223 13163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:22.254283 13163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.254364 13163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:22.254400 13163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:22.261193 13163 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39533
I20260925 14:55:22.261250 13291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39533 every 8 connection(s)
I20260925 14:55:22.261620 13163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:22.266458 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13163
I20260925 14:55:22.266541 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:22.266733 9781 external_mini_cluster.cc:1428] Setting key 7e7a5f80846bf9bbc6bbda0faac933ad
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[12788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:22.272107 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:22.262450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:59989 (local address 127.9.141.126:36901)
0925 14:55:22.262540 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:22.262543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:22.262709 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:55:22.263430 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:22.263432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:22.263444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:22.263498 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:22.264648 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.265196 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.266302 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.266563 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.266629 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:22.269237 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:22.269239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:22.269239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:22.269247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:22.270470 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:22.270954 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:22.270956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:22.270957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:22.271021 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:22.271381 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:22.271382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:22.271383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:22.271520 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:22.271600 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:22.271923 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 14:55:22.272013 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 14:55:22.272330 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:22.262357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36901 (local address 127.9.141.67:59989)
0925 14:55:22.262571 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:22.262624 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:22.263208 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:22.263576 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:22.263581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:22.264249 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:22.264544 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:22.264549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.265297 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.265298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:22.266183 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:22.266185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.266634 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.266635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:22.266774 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:22.267431 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:22.267441 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:22.269116 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:22.270574 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:22.270581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:22.270583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:22.270849 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:22.271096 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:22.271097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:22.271097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:22.271289 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:22.271628 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:22.271634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:22.271770 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:55:22.271884 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:55:22.272045 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260925 14:55:22.272897 13292 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36901
I20260925 14:55:22.273021 13292 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:22.273217 13292 heartbeater.cc:511] Master 127.9.141.126:36901 requested a full tablet report, sending...
I20260925 14:55:22.273581 12833 ts_manager.cc:194] Registered new tserver with Master: 35ff3555ea144d8dae6737fcd8c46504 (127.9.141.67:39533)
I20260925 14:55:22.273984 12833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:59989
I20260925 14:55:22.278551 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[12788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[12788](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348122, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:22.302356 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:22.294653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.9.141.126:36901)
0925 14:55:22.294765 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:22.294768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:22.294857 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:55:22.294956 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:22.294957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:22.294967 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:22.295056 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:22.295675 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.296232 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.297051 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:22.297253 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:22.297342 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:22.299627 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:22.299629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:22.299629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:22.299637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:22.301108 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:22.301526 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:22.301527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:22.301527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:22.301596 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:22.301801 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:22.301802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:22.301802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:22.301955 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:22.302059 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:22.302170 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:55:22.302248 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 14:55:22.302596 12833 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41778
I20260925 14:55:22.303095 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12895
I20260925 14:55:22.307981 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13029
I20260925 14:55:22.312346 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13163
I20260925 14:55:22.316670 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 12801
2026-09-25T14:55:22Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1978 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[13304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[13304](info): set up 2 sockets
Sep 25 14:55:22 dist-test-slave-1sv7 krb5kdc[13304](info): commencing operation
krb5kdc: starting...
W20260925 14:55:23.364319 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:23 dist-test-slave-1sv7 krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:23.492856 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37407
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:42437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:23.575675 13317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:23.575919 13317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:23.575958 13317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:23.577247 13317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:23.577307 13317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:23.577332 13317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:23.577375 13317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:23.577404 13317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:23.578856 13317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:42437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:23.579116 13317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:23.579387 13317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:23.584738 13322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.584746 13325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.584858 13323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:23.584993 13317 server_base.cc:1034] running on GCE node
I20260925 14:55:23.585253 13317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:23.585541 13317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:23.586714 13317 hybrid_clock.cc:648] HybridClock initialized: now 1790348123586600 us; error 134 us; skew 500 ppm
Sep 25 14:55:23 dist-test-slave-1sv7 krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:23.589300 13317 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:23.589792 13317 webserver.cc:460] Webserver started at http://127.9.141.126:36915/ using document root <none> and password file <none>
I20260925 14:55:23.590018 13317 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:23.590085 13317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:23.590235 13317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:23.591398 13317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "09994fe1c88242b383dc19bea86ab495"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "6af8d749a9fcdba7e94898d03a7751ae"
server_key_iv: "c3a32ccb08c492040a9400322d4c1294"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.591756 13317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "09994fe1c88242b383dc19bea86ab495"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "6af8d749a9fcdba7e94898d03a7751ae"
server_key_iv: "c3a32ccb08c492040a9400322d4c1294"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.593201 13317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:23.594069 13332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.594275 13317 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:23.594383 13317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "09994fe1c88242b383dc19bea86ab495"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "6af8d749a9fcdba7e94898d03a7751ae"
server_key_iv: "c3a32ccb08c492040a9400322d4c1294"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.594462 13317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:23.617868 13317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:23.620218 13317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:23.620407 13317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:23.624220 13317 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:37407
I20260925 14:55:23.624660 13317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:23.624666 13384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:37407 every 8 connection(s)
I20260925 14:55:23.625316 13385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:23.627741 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13317
I20260925 14:55:23.627785 13385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495: Bootstrap starting.
I20260925 14:55:23.627868 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:23.628067 9781 external_mini_cluster.cc:1428] Setting key 40d2fd6383d6f18dc362b2fa105d7b84
I20260925 14:55:23.628525 13385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:23.628919 13385 log.cc:824] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:23.629709 13385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495: No bootstrap required, opened a new log
I20260925 14:55:23.631896 13385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09994fe1c88242b383dc19bea86ab495" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37407 } }
I20260925 14:55:23.632061 13385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:23.632093 13385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09994fe1c88242b383dc19bea86ab495, State: Initialized, Role: FOLLOWER
I20260925 14:55:23.632236 13385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09994fe1c88242b383dc19bea86ab495" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37407 } }
I20260925 14:55:23.632354 13385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:23.632426 13385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:23.632478 13385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:23.633134 13385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09994fe1c88242b383dc19bea86ab495" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37407 } }
I20260925 14:55:23.633272 13385 leader_election.cc:302] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09994fe1c88242b383dc19bea86ab495; no voters:
I20260925 14:55:23.633443 13385 leader_election.cc:288] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:23.633543 13390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:23.633703 13385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:23.633720 13390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [term 1 LEADER]: Becoming Leader. State: Replica: 09994fe1c88242b383dc19bea86ab495, State: Running, Role: LEADER
I20260925 14:55:23.633812 13390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09994fe1c88242b383dc19bea86ab495" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37407 } }
I20260925 14:55:23.634148 13391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09994fe1c88242b383dc19bea86ab495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09994fe1c88242b383dc19bea86ab495" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37407 } } }
I20260925 14:55:23.634358 13391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:23.634279 13392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09994fe1c88242b383dc19bea86ab495. Latest consensus state: current_term: 1 leader_uuid: "09994fe1c88242b383dc19bea86ab495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09994fe1c88242b383dc19bea86ab495" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37407 } } }
I20260925 14:55:23.634476 13392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:23.634621 13399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:23.635107 13399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:55:23 dist-test-slave-1sv7 krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:23.636837 13399 catalog_manager.cc:1347] Generated new cluster ID: 2e5ef6c3f2444d11b324c22830015ba6
I20260925 14:55:23.636898 13399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:23.640700 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.628645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.9.141.126:37407)
0925 14:55:23.628808 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.628824 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.628849 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:55:23.629288 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.629295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.629342 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.629464 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:23.630211 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.631106 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.632203 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.632770 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.632855 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.636565 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:23.636569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:23.636569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:23.636581 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:23.638579 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.638995 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.638998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.638999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.639118 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.639346 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.639347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.639347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.639673 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:23.639833 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.640307 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.640494 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260925 14:55:23.680207 13399 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:23.681016 13399 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:23.687453 13399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09994fe1c88242b383dc19bea86ab495: Generated new TSK 0
I20260925 14:55:23.687649 13399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:23.719488 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37407
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:42437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:23.800679 13411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:23.800908 13411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:23.800946 13411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:23.802234 13411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:23.802305 13411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:23.802350 13411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:23.803918 13411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:42437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:23.804364 13411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:23.804637 13411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:23.810155 13416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.810127 13417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:23.810169 13411 server_base.cc:1034] running on GCE node
W20260925 14:55:23.810135 13419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:23.811841 13411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:23.812165 13411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:23.813339 13411 hybrid_clock.cc:648] HybridClock initialized: now 1790348123813310 us; error 48 us; skew 500 ppm
Sep 25 14:55:23 dist-test-slave-1sv7 krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:23.815874 13411 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:23.816313 13411 webserver.cc:460] Webserver started at http://127.9.141.65:46831/ using document root <none> and password file <none>
I20260925 14:55:23.816607 13411 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:23.816680 13411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:23.816830 13411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:23.817915 13411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "307a0c7d0e354e91928dff84f4a38c43"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "bdb135351a9501521fa744ef6644df1c"
server_key_iv: "3990f33cd0911679b8e45ac4b6c0801e"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.818265 13411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "307a0c7d0e354e91928dff84f4a38c43"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "bdb135351a9501521fa744ef6644df1c"
server_key_iv: "3990f33cd0911679b8e45ac4b6c0801e"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.819680 13411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:23.820607 13426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.820787 13411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:23.820885 13411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "307a0c7d0e354e91928dff84f4a38c43"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "bdb135351a9501521fa744ef6644df1c"
server_key_iv: "3990f33cd0911679b8e45ac4b6c0801e"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.820972 13411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:23.841822 13411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:23.843793 13411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:23.843956 13411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:23.844197 13411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:23.844594 13411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:23.844650 13411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.844686 13411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:23.844719 13411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.850440 13411 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:35483
I20260925 14:55:23.850479 13539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:35483 every 8 connection(s)
I20260925 14:55:23.850843 13411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:23.855036 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13411
I20260925 14:55:23.855140 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:23.855316 9781 external_mini_cluster.cc:1428] Setting key 979b1f1f30bf2b78358d6ec54c6ef536
Sep 25 14:55:23 dist-test-slave-1sv7 krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:23.861424 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.851691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:40655 (local address 127.9.141.126:37407)
0925 14:55:23.851748 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:23.851750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:23.851842 (+ 92us) server_negotiation.cc:407] Connection header received
0925 14:55:23.852858 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:23.852860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:23.852870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:23.852938 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:23.853548 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.854075 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.855261 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.855557 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.855628 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.858374 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:23.858376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:23.858377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:23.858386 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:23.859936 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.860422 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.860424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.860424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.860487 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:23.860758 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:23.860759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:23.860760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:23.860884 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:23.860963 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:23.861241 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:55:23.861335 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:55:23.861640 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:23.851575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37407 (local address 127.9.141.65:40655)
0925 14:55:23.851729 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:23.851772 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:23.852757 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:23.852996 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:23.852999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:23.853253 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:23.853440 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.853444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.854165 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.854165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.855156 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:23.855159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:23.855706 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:23.855708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:23.855863 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:23.856434 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:23.856445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:23.858274 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:23.860064 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.860071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.860073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.860302 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.860566 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:23.860566 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:23.860567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:23.860680 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:23.860958 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:23.860965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:23.861106 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:55:23.861207 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:55:23.861374 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260925 14:55:23.862100 13540 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37407
I20260925 14:55:23.862215 13540 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:23.862396 13540 heartbeater.cc:511] Master 127.9.141.126:37407 requested a full tablet report, sending...
I20260925 14:55:23.862905 13349 ts_manager.cc:194] Registered new tserver with Master: 307a0c7d0e354e91928dff84f4a38c43 (127.9.141.65:35483)
I20260925 14:55:23.863543 13349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:40655
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:23.881377 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37407
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:42437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:23.972759 13545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:23.972987 13545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:23.973026 13545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:23.975167 13545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:23.975243 13545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:23.975301 13545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:23.977816 13545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:42437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:23.978099 13545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:23.978384 13545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:23.985266 13553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.985299 13550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:23.985299 13551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:23.988468 13545 server_base.cc:1034] running on GCE node
I20260925 14:55:23.988714 13545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:23.989006 13545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:23.990165 13545 hybrid_clock.cc:648] HybridClock initialized: now 1790348123990150 us; error 30 us; skew 500 ppm
Sep 25 14:55:23 dist-test-slave-1sv7 krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:23.993734 13545 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:23.994366 13545 webserver.cc:460] Webserver started at http://127.9.141.66:38061/ using document root <none> and password file <none>
I20260925 14:55:23.994659 13545 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:23.994725 13545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:23.994884 13545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:23.996573 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "65db86d37f7c4668a71da36e9bb0b0c2"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "fb0230a18334576c7a4f5325dee2a65b"
server_key_iv: "26baf50d3cb0c20be932f406d58f0183"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.997087 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "65db86d37f7c4668a71da36e9bb0b0c2"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "fb0230a18334576c7a4f5325dee2a65b"
server_key_iv: "26baf50d3cb0c20be932f406d58f0183"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.998764 13545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:55:23.999588 13560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:23.999761 13545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:23.999861 13545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "65db86d37f7c4668a71da36e9bb0b0c2"
format_stamp: "Formatted at 2026-09-25 14:55:23 on dist-test-slave-1sv7"
server_key: "fb0230a18334576c7a4f5325dee2a65b"
server_key_iv: "26baf50d3cb0c20be932f406d58f0183"
server_key_version: "encryptionkey@0"
I20260925 14:55:23.999957 13545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:24.028527 13545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:24.031075 13545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:24.031261 13545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:24.031533 13545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:24.031867 13545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:24.031926 13545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:24.031971 13545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:24.032004 13545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:24.038698 13545 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:35649
I20260925 14:55:24.038749 13673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:35649 every 8 connection(s)
I20260925 14:55:24.039072 13545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348123, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:24.046967 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13545
I20260925 14:55:24.047116 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:24.047339 9781 external_mini_cluster.cc:1428] Setting key d1281a8ba91e7d465065790ff4c88c71
I20260925 14:55:24.048864 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.039879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:39937 (local address 127.9.141.126:37407)
0925 14:55:24.039927 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.039930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.040109 (+ 179us) server_negotiation.cc:407] Connection header received
0925 14:55:24.041057 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.041058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.041068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.041152 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.041713 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.042265 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.043283 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.043496 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.043556 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.045683 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.045685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.045685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.045691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.046976 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.047628 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.047629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.047629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.047722 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.048054 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.048055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.048055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.048192 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.048283 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.048672 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.048766 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:55:24.049101 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.039853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37407 (local address 127.9.141.66:39937)
0925 14:55:24.040008 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.040041 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:24.040965 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:24.041210 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:24.041213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:24.041445 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:24.041633 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.041637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.042361 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.042362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.043199 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.043201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.043582 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.043583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.043681 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.044174 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:24.044182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:24.045557 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:24.047102 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:24.047111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:24.047113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:24.047455 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:24.047805 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:24.047806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:24.047806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:24.047942 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:24.048273 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:24.048280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:24.048497 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 14:55:24.048647 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 14:55:24.048826 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260925 14:55:24.049607 13674 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37407
I20260925 14:55:24.049721 13674 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:24.049928 13674 heartbeater.cc:511] Master 127.9.141.126:37407 requested a full tablet report, sending...
I20260925 14:55:24.050315 13349 ts_manager.cc:194] Registered new tserver with Master: 65db86d37f7c4668a71da36e9bb0b0c2 (127.9.141.66:35649)
I20260925 14:55:24.050748 13349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:39937
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:24.075366 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37407
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:42437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:24.155674 13679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:24.156184 13679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:24.156210 13679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:24.157498 13679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:24.157543 13679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:24.157572 13679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:24.159121 13679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:42437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:24.159354 13679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:24.159576 13679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:24.164541 13685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:24.164535 13687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:24.164515 13684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:24.164796 13679 server_base.cc:1034] running on GCE node
I20260925 14:55:24.165023 13679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:24.165213 13679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:24.166373 13679 hybrid_clock.cc:648] HybridClock initialized: now 1790348124166363 us; error 30 us; skew 500 ppm
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348124, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:24.169034 13679 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:24.169565 13679 webserver.cc:460] Webserver started at http://127.9.141.67:34533/ using document root <none> and password file <none>
I20260925 14:55:24.169772 13679 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:24.169821 13679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:24.169909 13679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:24.170899 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "164d669333c0472f8544c578b37b175d"
format_stamp: "Formatted at 2026-09-25 14:55:24 on dist-test-slave-1sv7"
server_key: "2be9a2b886914055e2539529dd9cbe5e"
server_key_iv: "c4b4681be620d260eb194de98c66a80c"
server_key_version: "encryptionkey@0"
I20260925 14:55:24.171203 13679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "164d669333c0472f8544c578b37b175d"
format_stamp: "Formatted at 2026-09-25 14:55:24 on dist-test-slave-1sv7"
server_key: "2be9a2b886914055e2539529dd9cbe5e"
server_key_iv: "c4b4681be620d260eb194de98c66a80c"
server_key_version: "encryptionkey@0"
I20260925 14:55:24.172511 13679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:24.173345 13694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:24.173611 13679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:24.173719 13679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "164d669333c0472f8544c578b37b175d"
format_stamp: "Formatted at 2026-09-25 14:55:24 on dist-test-slave-1sv7"
server_key: "2be9a2b886914055e2539529dd9cbe5e"
server_key_iv: "c4b4681be620d260eb194de98c66a80c"
server_key_version: "encryptionkey@0"
I20260925 14:55:24.173800 13679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:24.196146 13679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:24.198725 13679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:24.198889 13679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:24.199127 13679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:24.199440 13679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:24.199496 13679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:24.199541 13679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:24.199580 13679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:24.205484 13679 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:33673
I20260925 14:55:24.205530 13807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:33673 every 8 connection(s)
I20260925 14:55:24.205842 13679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:24.210258 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13679
I20260925 14:55:24.210333 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:24.210546 9781 external_mini_cluster.cc:1428] Setting key 01c38892acbb6a7fc879bf03f7b69474
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348124, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:24.215788 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:38569 (local address 127.9.141.126:37407)
0925 14:55:24.206647 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.206649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.206851 (+ 202us) server_negotiation.cc:407] Connection header received
0925 14:55:24.207799 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.207801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.207813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.207870 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.208525 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.209032 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.210094 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.210326 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.210396 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.213096 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.213098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.213098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.213104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.214460 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.214869 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.214870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.214870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.214926 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.215192 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.215193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.215193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.215306 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.215370 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.215652 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.215724 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:24.215973 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.206577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37407 (local address 127.9.141.67:38569)
0925 14:55:24.206731 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:24.206766 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:24.207683 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:24.207962 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:24.207971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:24.208218 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:24.208427 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.208431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.209157 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.209158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.209972 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:24.209974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.210500 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.210502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:24.210664 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.211323 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:24.211333 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:24.212993 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:24.214555 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:24.214560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:24.214562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:24.214787 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:24.214998 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:24.214999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:24.214999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:24.215104 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:24.215390 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:24.215395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:24.215517 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 14:55:24.215624 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:55:24.215747 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260925 14:55:24.216557 13808 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37407
I20260925 14:55:24.216692 13808 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:24.216888 13808 heartbeater.cc:511] Master 127.9.141.126:37407 requested a full tablet report, sending...
I20260925 14:55:24.217324 13349 ts_manager.cc:194] Registered new tserver with Master: 164d669333c0472f8544c578b37b175d (127.9.141.67:33673)
I20260925 14:55:24.217711 13349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:38569
I20260925 14:55:24.222502 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:24.246711 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.238871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.9.141.65:35483)
0925 14:55:24.239058 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.239062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.239074 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:24.239100 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.239102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.239116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.239192 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.239795 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.240595 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.241346 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.241662 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.241720 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.243930 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.243937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.243938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.243944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.245573 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.245937 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.245940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.245941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.246004 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.246241 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.246242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.246242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.246362 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.246461 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.246578 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.246634 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:24.254900 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:24.247947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.9.141.126:37407)
0925 14:55:24.248000 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:24.248002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:24.248014 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:24.248090 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:24.248091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:24.248107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:24.248169 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:24.248716 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.249229 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.249920 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:24.250106 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:24.250208 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:24.252560 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:24.252561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:24.252562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:24.252569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:24.253730 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.254064 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.254065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.254066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.254122 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:24.254332 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:24.254333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:24.254334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:24.254476 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:24.254580 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:24.254698 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:55:24.254797 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:55:24.255195 13349 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50984
I20260925 14:55:24.255596 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13411
I20260925 14:55:24.260178 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13545
I20260925 14:55:24.264663 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13679
I20260925 14:55:24.269428 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13317
2026-09-25T14:55:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1953 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13820](info): set up 2 sockets
Sep 25 14:55:24 dist-test-slave-1sv7 krb5kdc[13820](info): commencing operation
krb5kdc: starting...
W20260925 14:55:25.324656 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:25.445508 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44729
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:44729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:25.525200 13833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:25.525406 13833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:25.525430 13833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:25.526511 13833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:25.526546 13833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:25.526559 13833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:25.526571 13833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:25.526584 13833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:25.527977 13833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:44729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:25.528159 13833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:25.528324 13833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:25.533306 13838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.533324 13839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.533473 13841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:25.533779 13833 server_base.cc:1034] running on GCE node
I20260925 14:55:25.534111 13833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:25.534463 13833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:25.535658 13833 hybrid_clock.cc:648] HybridClock initialized: now 1790348125535663 us; error 70 us; skew 500 ppm
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:25.539180 13833 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:25.539602 13833 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:25.539672 13833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:25.539816 13833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:25.541289 13833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "12f6cc9329a7426e9d31cbe7c3821e83"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "5455ef2a7c0b926f326da120064c7f49"
server_key_iv: "87d9f6d112f62636fb9b92187725bdca"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.541731 13833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "12f6cc9329a7426e9d31cbe7c3821e83"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "5455ef2a7c0b926f326da120064c7f49"
server_key_iv: "87d9f6d112f62636fb9b92187725bdca"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.543442 13833 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:55:25.544216 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.544452 13833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:25.544550 13833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "12f6cc9329a7426e9d31cbe7c3821e83"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "5455ef2a7c0b926f326da120064c7f49"
server_key_iv: "87d9f6d112f62636fb9b92187725bdca"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.544638 13833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:25.563467 13833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:25.565553 13833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:25.565723 13833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:25.569730 13833 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:44729
I20260925 14:55:25.569808 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:44729 every 8 connection(s)
I20260925 14:55:25.570139 13833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:25.571074 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:25.573522 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83: Bootstrap starting.
I20260925 14:55:25.574103 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:25.574409 13900 log.cc:824] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:25.575065 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83: No bootstrap required, opened a new log
I20260925 14:55:25.576777 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44729 } }
I20260925 14:55:25.576913 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:25.576936 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f6cc9329a7426e9d31cbe7c3821e83, State: Initialized, Role: FOLLOWER
I20260925 14:55:25.577111 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44729 } }
I20260925 14:55:25.577175 13900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:25.577239 13900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:25.577288 13900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:25.578058 13900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44729 } }
I20260925 14:55:25.578172 13900 leader_election.cc:302] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f6cc9329a7426e9d31cbe7c3821e83; no voters:
I20260925 14:55:25.578372 13900 leader_election.cc:288] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:25.578421 13903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:25.578603 13903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [term 1 LEADER]: Becoming Leader. State: Replica: 12f6cc9329a7426e9d31cbe7c3821e83, State: Running, Role: LEADER
I20260925 14:55:25.578634 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:25.578713 13903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44729 } }
I20260925 14:55:25.579125 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f6cc9329a7426e9d31cbe7c3821e83. Latest consensus state: current_term: 1 leader_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44729 } } }
I20260925 14:55:25.579241 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:25.579115 13904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f6cc9329a7426e9d31cbe7c3821e83" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44729 } } }
I20260925 14:55:25.579440 13904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:25.579670 13912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:25.580188 13912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:25.580216 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13833
I20260925 14:55:25.580304 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:25.580533 9781 external_mini_cluster.cc:1428] Setting key 7e7fc5005621b84518478b0a2c665563
I20260925 14:55:25.581987 13912 catalog_manager.cc:1347] Generated new cluster ID: a018fae53f814490b79f3ba0d1cead97
I20260925 14:55:25.582037 13912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:25.592177 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.581152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.9.141.126:44729)
0925 14:55:25.581290 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:25.581299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:25.581329 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:55:25.581448 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:25.581473 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:25.581533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:25.581671 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:25.582470 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.583971 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.585018 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.585693 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.585808 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:25.588637 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:25.588640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:25.588641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:25.588653 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:25.590571 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:25.590981 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:25.590984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:25.590985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:25.591084 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:25.591291 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:25.591291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:25.591292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:25.591571 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:25.591718 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:25.591826 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:55:25.591959 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260925 14:55:25.598073 13912 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:25.598589 13912 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:25.604794 13912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83: Generated new TSK 0
I20260925 14:55:25.604981 13912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:25.635864 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44729
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:25.722684 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:25.722884 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:25.722908 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:25.723994 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:25.724027 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:25.724053 13926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:25.725625 13926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:25.725824 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:25.726009 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:25.731201 13934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:25.731310 13926 server_base.cc:1034] running on GCE node
W20260925 14:55:25.731252 13932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.731199 13931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:25.731791 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:25.732030 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:25.733177 13926 hybrid_clock.cc:648] HybridClock initialized: now 1790348125733166 us; error 33 us; skew 500 ppm
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:25.735829 13926 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:25.736125 13926 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:25.736168 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:25.736256 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:25.737376 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "c939f779969d413295046fa62887897e"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "8f1c3990c2d009a94ccf24b42ef15cc1"
server_key_iv: "b40a3a3c7ebb480b74394d3f6290a7d4"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.737665 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "c939f779969d413295046fa62887897e"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "8f1c3990c2d009a94ccf24b42ef15cc1"
server_key_iv: "b40a3a3c7ebb480b74394d3f6290a7d4"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.738938 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:25.739634 13940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.739833 13926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:25.739948 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "c939f779969d413295046fa62887897e"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "8f1c3990c2d009a94ccf24b42ef15cc1"
server_key_iv: "b40a3a3c7ebb480b74394d3f6290a7d4"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.740027 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:25.755884 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:25.758293 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:25.758467 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:25.758701 13926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:25.759033 13926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:25.759088 13926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.759130 13926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:25.759157 13926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.764833 13926 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:46321
I20260925 14:55:25.764870 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:46321 every 8 connection(s)
I20260925 14:55:25.765157 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:25.771173 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 13926
I20260925 14:55:25.771310 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:25.771530 9781 external_mini_cluster.cc:1428] Setting key a53613bae8fa238366e50e9e04db76eb
I20260925 14:55:25.775101 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.765875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44729 (local address 127.9.141.65:38243)
0925 14:55:25.766052 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:25.766101 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:25.766638 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:25.766902 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:25.766907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:25.767170 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:25.767662 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:25.767667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.768426 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.768427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:25.769243 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:25.769245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.769670 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.769671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:25.769795 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:25.770282 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:25.770290 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:25.771945 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:25.773659 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:25.773668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:25.773671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:25.773939 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:25.774154 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:25.774155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:25.774155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:25.774256 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:25.774556 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:25.774562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:25.774682 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:55:25.774784 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:55:25.774932 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260925 14:55:25.775097 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.765899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:38243 (local address 127.9.141.126:44729)
0925 14:55:25.766010 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:25.766012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:25.766190 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:55:25.766766 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:25.766768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:25.766780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:25.766832 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:25.767773 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.768302 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.769341 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.769572 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.769659 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:25.772041 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:25.772042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:25.772042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:25.772049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:25.773560 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:25.774013 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:25.774015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:25.774015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:25.774079 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:25.774324 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:25.774324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:25.774325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:25.774460 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:25.774532 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:25.774827 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 14:55:25.774925 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 14:55:25.776015 14054 heartbeater.cc:347] Connected to a master server at 127.9.141.126:44729
I20260925 14:55:25.776129 14054 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:25.776280 14054 heartbeater.cc:511] Master 127.9.141.126:44729 requested a full tablet report, sending...
I20260925 14:55:25.776813 13864 ts_manager.cc:194] Registered new tserver with Master: c939f779969d413295046fa62887897e (127.9.141.65:46321)
I20260925 14:55:25.777458 13864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:38243
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:25.800092 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44729
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:25.883293 14059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:25.883513 14059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:25.883538 14059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:25.884757 14059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:25.884799 14059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:25.884827 14059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:25.886345 14059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:25.886585 14059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:25.886837 14059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:55:25.891912 14059 server_base.cc:1034] running on GCE node
W20260925 14:55:25.891959 14065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.891996 14067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:25.891992 14064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:25.892257 14059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:25.892542 14059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:25.893781 14059 hybrid_clock.cc:648] HybridClock initialized: now 1790348125893774 us; error 37 us; skew 500 ppm
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:25.896338 14059 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:25.896723 14059 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:25.896771 14059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:25.896862 14059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:25.897992 14059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "9f95e816707a432a8d9d266bcaf5ad05"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "6ca2482522f88ee4eb3683c33b221150"
server_key_iv: "df1379321e10331f5b6e7c8e8b1d5b90"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.898296 14059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "9f95e816707a432a8d9d266bcaf5ad05"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "6ca2482522f88ee4eb3683c33b221150"
server_key_iv: "df1379321e10331f5b6e7c8e8b1d5b90"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.899533 14059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:55:25.900220 14073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.900434 14059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:25.900547 14059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "9f95e816707a432a8d9d266bcaf5ad05"
format_stamp: "Formatted at 2026-09-25 14:55:25 on dist-test-slave-1sv7"
server_key: "6ca2482522f88ee4eb3683c33b221150"
server_key_iv: "df1379321e10331f5b6e7c8e8b1d5b90"
server_key_version: "encryptionkey@0"
I20260925 14:55:25.900615 14059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:25.922088 14059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:25.924489 14059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:25.924602 14059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:25.924835 14059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:25.925215 14059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:25.925271 14059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.925314 14059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:25.925340 14059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:25.931331 14059 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:38957
I20260925 14:55:25.931370 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:38957 every 8 connection(s)
I20260925 14:55:25.931697 14059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:25.935460 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14059
I20260925 14:55:25.935571 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:25.935745 9781 external_mini_cluster.cc:1428] Setting key 4688620f08d2a4cec11ca9e911083b7a
Sep 25 14:55:25 dist-test-slave-1sv7 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348125, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:25.941870 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.932550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:49189 (local address 127.9.141.126:44729)
0925 14:55:25.932655 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:25.932657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:25.932877 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:55:25.933430 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:25.933431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:25.933441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:25.933504 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:25.934390 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.934922 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.936098 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.936413 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.936484 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:25.938932 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:25.938933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:25.938933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:25.938939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:25.940365 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:25.940899 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:25.940900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:25.940900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:25.940949 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:25.941243 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:25.941243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:25.941244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:25.941338 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:25.941385 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:25.941744 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:55:25.941807 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 14:55:25.942153 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:25.932567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44729 (local address 127.9.141.66:49189)
0925 14:55:25.932729 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:25.932772 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:25.933331 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:25.933561 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:25.933564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:25.933820 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:25.934310 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:25.934315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.935008 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.935008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:25.935953 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:25.935956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:25.936496 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:25.936497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:25.936666 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:25.937197 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:25.937208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:25.938842 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:25.940470 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:25.940477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:25.940479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:25.940817 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:25.941017 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:25.941018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:25.941019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:25.941168 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:25.941403 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:25.941409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:25.941588 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:55:25.941727 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 14:55:25.941910 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260925 14:55:25.942656 14187 heartbeater.cc:347] Connected to a master server at 127.9.141.126:44729
I20260925 14:55:25.942770 14187 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:25.942927 14187 heartbeater.cc:511] Master 127.9.141.126:44729 requested a full tablet report, sending...
I20260925 14:55:25.943353 13864 ts_manager.cc:194] Registered new tserver with Master: 9f95e816707a432a8d9d266bcaf5ad05 (127.9.141.66:38957)
I20260925 14:55:25.943739 13864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:49189
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:25.963948 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44729
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:26.044828 14192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.045025 14192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.045050 14192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.046131 14192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.046166 14192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.046200 14192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:26.047636 14192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:26.047861 14192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.048110 14192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.053074 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.053086 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.053263 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.053280 14192 server_base.cc:1034] running on GCE node
I20260925 14:55:26.053547 14192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.053740 14192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.054903 14192 hybrid_clock.cc:648] HybridClock initialized: now 1790348126054896 us; error 36 us; skew 500 ppm
Sep 25 14:55:26 dist-test-slave-1sv7 krb5kdc[13820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348126, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:26.057411 14192 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:26.057789 14192 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.057837 14192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.057925 14192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:26.058969 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "61dec72e8c1b4a7aa06daa20f27bbc59"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "65da8c30c4f7de6c1965c73439233937"
server_key_iv: "29280b16d06922b7e9ae37bc92a3b564"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.059268 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "61dec72e8c1b4a7aa06daa20f27bbc59"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "65da8c30c4f7de6c1965c73439233937"
server_key_iv: "29280b16d06922b7e9ae37bc92a3b564"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.060564 14192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:26.061290 14206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.061486 14192 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:26.061570 14192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "61dec72e8c1b4a7aa06daa20f27bbc59"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "65da8c30c4f7de6c1965c73439233937"
server_key_iv: "29280b16d06922b7e9ae37bc92a3b564"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.061621 14192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.077894 14192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.080569 14192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.080683 14192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.080869 14192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:26.081110 14192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:26.081139 14192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.081161 14192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:26.081173 14192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.086978 14192 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:34627
I20260925 14:55:26.087061 14319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:34627 every 8 connection(s)
I20260925 14:55:26.087312 14192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:26.088722 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14192
I20260925 14:55:26.088840 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:26.089073 9781 external_mini_cluster.cc:1428] Setting key 4ff0a61aeeddf446334fed1e1309131d
Sep 25 14:55:26 dist-test-slave-1sv7 krb5kdc[13820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348126, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:26.097246 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.088051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:42431 (local address 127.9.141.126:44729)
0925 14:55:26.088136 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.088138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.088477 (+ 339us) server_negotiation.cc:407] Connection header received
0925 14:55:26.089042 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.089044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.089058 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.089137 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.090363 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.090847 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.091937 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.092149 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.092210 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.094422 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.094423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.094424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:26.094430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.095846 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:26.096245 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:26.096245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:26.096246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:26.096294 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:26.096682 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:26.096683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:26.096683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:26.096801 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.096859 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.097111 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.097179 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:55:26.097407 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.088015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44729 (local address 127.9.141.67:42431)
0925 14:55:26.088239 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.088288 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.088920 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.089249 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.089254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.089664 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.090283 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.090288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.090943 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.090943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.091843 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.091845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.092256 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.092258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.092464 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.092927 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:26.092936 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.094337 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.095944 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:26.095949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:26.095951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:26.096171 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:26.096373 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:26.096374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:26.096374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:26.096559 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:26.096866 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.096871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.096991 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.097067 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.097189 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":124,"threads_started":1}
I20260925 14:55:26.097998 14320 heartbeater.cc:347] Connected to a master server at 127.9.141.126:44729
I20260925 14:55:26.098080 14320 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:26.098245 14320 heartbeater.cc:511] Master 127.9.141.126:44729 requested a full tablet report, sending...
I20260925 14:55:26.098623 13864 ts_manager.cc:194] Registered new tserver with Master: 61dec72e8c1b4a7aa06daa20f27bbc59 (127.9.141.67:34627)
I20260925 14:55:26.098990 13864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:42431
I20260925 14:55:26.100796 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:26.108072 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.102157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.9.141.126:44729)
0925 14:55:26.102237 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.102239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.102350 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:55:26.102445 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.102446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.102455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.102500 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.103034 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.103567 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.104290 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.104571 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.104666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.105796 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.105798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.105798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:26.105805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.106903 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:26.107258 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:26.107259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:26.107260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:26.107311 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:26.107520 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:26.107521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:26.107521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:26.107669 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.107792 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.107901 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.107975 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:55:26.109395 13864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:26.110970 13864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:26.118050 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.113395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:38957 (local address 127.0.0.1:39806)
0925 14:55:26.113727 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.113768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.113841 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.114209 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.114210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.114213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:26.114284 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.114286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.115173 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.115174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.116413 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.116416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.117672 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.117673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.117768 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.117785 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.117878 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.117942 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":7,"thread_start_us":143,"threads_started":1}
I20260925 14:55:26.118059 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.113567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.9.141.66:38957)
0925 14:55:26.113698 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.113704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.113933 (+ 229us) server_negotiation.cc:407] Connection header received
0925 14:55:26.114025 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.114028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.114045 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.114120 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:26.114355 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.115060 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.116491 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.117553 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.117653 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.117739 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.117904 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.117943 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260925 14:55:26.119159 14119 tablet_service.cc:1467] Processing CreateTablet for tablet fbf4eaa77d9a4d9394349d4f27245748 (DEFAULT_TABLE table=test-table [id=2b0647a76bfe49528bf50875b4fc9404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:26.119616 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf4eaa77d9a4d9394349d4f27245748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.122512 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.113395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:34627 (local address 127.0.0.1:45390)
0925 14:55:26.113726 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.113766 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.113840 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.114236 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.114237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.114238 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:26.114275 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.114276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.115573 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.115574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.120986 (+ 5412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.120990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.122213 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.122214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.122297 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.122301 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.122391 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.122437 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260925 14:55:26.122850 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.113469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.9.141.67:34627)
0925 14:55:26.113980 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.113983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.114014 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:55:26.114051 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.114053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.114068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.114135 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:26.114354 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.115461 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.121113 (+ 5652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.121990 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.122112 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.122175 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.122676 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.122723 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":93,"threads_started":1}
I20260925 14:55:26.123493 14254 tablet_service.cc:1467] Processing CreateTablet for tablet fbf4eaa77d9a4d9394349d4f27245748 (DEFAULT_TABLE table=test-table [id=2b0647a76bfe49528bf50875b4fc9404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:26.123831 14254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf4eaa77d9a4d9394349d4f27245748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.123942 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.113593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46321 (local address 127.0.0.1:41120)
0925 14:55:26.113866 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.113875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.113929 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.114204 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.114207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.114210 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:26.114271 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.114273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.121088 (+ 6815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.121089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.122502 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.122505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.123655 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.123656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.123744 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.123749 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.123800 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.123848 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":80,"threads_started":1}
I20260925 14:55:26.124275 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.113706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.9.141.65:46321)
0925 14:55:26.113873 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.113877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.113922 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:55:26.113997 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.113999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.114016 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.114095 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:26.114355 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.115461 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.122615 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.123477 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.123555 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.123617 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.124128 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.124178 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260925 14:55:26.124905 13988 tablet_service.cc:1467] Processing CreateTablet for tablet fbf4eaa77d9a4d9394349d4f27245748 (DEFAULT_TABLE table=test-table [id=2b0647a76bfe49528bf50875b4fc9404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:26.125267 13988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf4eaa77d9a4d9394349d4f27245748. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.126602 14338 tablet_bootstrap.cc:492] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Bootstrap starting.
I20260925 14:55:26.127427 14338 tablet_bootstrap.cc:654] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.127977 14338 log.cc:824] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.128546 14337 tablet_bootstrap.cc:492] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Bootstrap starting.
I20260925 14:55:26.128774 14338 tablet_bootstrap.cc:492] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: No bootstrap required, opened a new log
I20260925 14:55:26.128849 14338 ts_tablet_manager.cc:1397] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:26.128873 14340 tablet_bootstrap.cc:492] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e: Bootstrap starting.
I20260925 14:55:26.129206 14337 tablet_bootstrap.cc:654] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.129525 14340 tablet_bootstrap.cc:654] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.129760 14337 log.cc:824] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.130151 14340 log.cc:824] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.130546 14337 tablet_bootstrap.cc:492] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: No bootstrap required, opened a new log
I20260925 14:55:26.130622 14337 ts_tablet_manager.cc:1397] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:26.130893 14338 raft_consensus.cc:348] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.131080 14338 raft_consensus.cc:374] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.131125 14338 raft_consensus.cc:729] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61dec72e8c1b4a7aa06daa20f27bbc59, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.131402 14338 consensus_queue.cc:260] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.131699 14338 ts_tablet_manager.cc:1428] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:26.131841 14320 heartbeater.cc:503] Master 127.9.141.126:44729 was elected leader, sending a full tablet report...
I20260925 14:55:26.132980 14337 raft_consensus.cc:348] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.133167 14337 raft_consensus.cc:374] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.133204 14337 raft_consensus.cc:729] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f95e816707a432a8d9d266bcaf5ad05, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.133318 14337 consensus_queue.cc:260] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.133585 14337 ts_tablet_manager.cc:1428] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:26.133620 14187 heartbeater.cc:503] Master 127.9.141.126:44729 was elected leader, sending a full tablet report...
I20260925 14:55:26.133934 14340 tablet_bootstrap.cc:492] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e: No bootstrap required, opened a new log
I20260925 14:55:26.134008 14340 ts_tablet_manager.cc:1397] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:55:26.135536 14340 raft_consensus.cc:348] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.135661 14340 raft_consensus.cc:374] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.135682 14340 raft_consensus.cc:729] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c939f779969d413295046fa62887897e, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.135794 14340 consensus_queue.cc:260] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.136083 14340 ts_tablet_manager.cc:1428] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:26.136121 14054 heartbeater.cc:503] Master 127.9.141.126:44729 was elected leader, sending a full tablet report...
I20260925 14:55:26.163403 14345 raft_consensus.cc:482] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:26.163527 14345 raft_consensus.cc:504] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.163923 14345 leader_election.cc:288] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61dec72e8c1b4a7aa06daa20f27bbc59 (127.9.141.67:34627), 9f95e816707a432a8d9d266bcaf5ad05 (127.9.141.66:38957)
I20260925 14:55:26.167714 14056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.164081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:34627 (local address 127.9.141.65:58433)
0925 14:55:26.164122 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.164135 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.164180 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.164367 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.164368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.164370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:26.164474 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.164476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.165510 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.165511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.166873 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.166875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.167437 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.167438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.167529 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.167532 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.167604 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.167643 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 14:55:26.167794 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.164109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:58433 (local address 127.9.141.67:34627)
0925 14:55:26.164150 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.164153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.164168 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:55:26.164239 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.164240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.164249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.164311 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:26.164610 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.165434 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.166970 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.167341 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.167438 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.167448 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.167610 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.167721 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:26.168049 14274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf4eaa77d9a4d9394349d4f27245748" candidate_uuid: "c939f779969d413295046fa62887897e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" is_pre_election: true
I20260925 14:55:26.168239 14274 raft_consensus.cc:2454] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c939f779969d413295046fa62887897e in term 0.
I20260925 14:55:26.168509 13943 leader_election.cc:302] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61dec72e8c1b4a7aa06daa20f27bbc59, c939f779969d413295046fa62887897e; no voters:
I20260925 14:55:26.168674 14345 raft_consensus.cc:2781] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:26.168726 14345 raft_consensus.cc:482] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:26.168756 14345 raft_consensus.cc:3037] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.169400 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.164291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:59369 (local address 127.9.141.66:38957)
0925 14:55:26.164333 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.164335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.164344 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:26.164449 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.164450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.164459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.164515 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:26.164681 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.165427 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.168415 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.168988 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.169068 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.169076 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.169279 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.169322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:26.169406 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.164090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:38957 (local address 127.9.141.65:59369)
0925 14:55:26.164302 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.164311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.164350 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.164516 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.164517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.164519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:26.164595 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.164596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.166975 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.166976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.168317 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.168319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.169101 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.169103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.169186 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.169191 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.169255 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.169303 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":14,"thread_start_us":117,"threads_started":1}
I20260925 14:55:26.169710 14345 raft_consensus.cc:504] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.170003 14141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf4eaa77d9a4d9394349d4f27245748" candidate_uuid: "c939f779969d413295046fa62887897e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f95e816707a432a8d9d266bcaf5ad05" is_pre_election: true
I20260925 14:55:26.170154 14141 raft_consensus.cc:2454] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c939f779969d413295046fa62887897e in term 0.
I20260925 14:55:26.170318 14274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf4eaa77d9a4d9394349d4f27245748" candidate_uuid: "c939f779969d413295046fa62887897e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59"
I20260925 14:55:26.170394 14274 raft_consensus.cc:3037] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.171084 14274 raft_consensus.cc:2454] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c939f779969d413295046fa62887897e in term 1.
I20260925 14:55:26.171218 14345 leader_election.cc:288] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [CANDIDATE]: Term 1 election: Requested vote from peers 61dec72e8c1b4a7aa06daa20f27bbc59 (127.9.141.67:34627), 9f95e816707a432a8d9d266bcaf5ad05 (127.9.141.66:38957)
I20260925 14:55:26.171278 13943 leader_election.cc:302] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61dec72e8c1b4a7aa06daa20f27bbc59, c939f779969d413295046fa62887897e; no voters:
I20260925 14:55:26.171375 14141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf4eaa77d9a4d9394349d4f27245748" candidate_uuid: "c939f779969d413295046fa62887897e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f95e816707a432a8d9d266bcaf5ad05"
I20260925 14:55:26.171449 14141 raft_consensus.cc:3037] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.172106 14141 raft_consensus.cc:2454] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c939f779969d413295046fa62887897e in term 1.
I20260925 14:55:26.172222 14345 raft_consensus.cc:2781] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:26.172416 14345 raft_consensus.cc:686] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [term 1 LEADER]: Becoming Leader. State: Replica: c939f779969d413295046fa62887897e, State: Running, Role: LEADER
I20260925 14:55:26.172518 14345 consensus_queue.cc:237] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } }
I20260925 14:55:26.173385 13864 catalog_manager.cc:5495] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e reported cstate change: term changed from 0 to 1, leader changed from <none> to c939f779969d413295046fa62887897e (127.9.141.65). New cstate: current_term: 1 leader_uuid: "c939f779969d413295046fa62887897e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c939f779969d413295046fa62887897e" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:26.180354 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.177228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.9.141.65:46321)
0925 14:55:26.177272 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.177275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.177323 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:55:26.177388 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.177389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.177408 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.177483 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:26.177588 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.178352 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.179321 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.179647 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.179722 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.179914 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:26.180056 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:26.180175 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.180242 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.180265 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:26.182178 14141 raft_consensus.cc:1261] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 1 FOLLOWER]: Refusing update from remote peer c939f779969d413295046fa62887897e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:26.182224 14274 raft_consensus.cc:1261] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 1 FOLLOWER]: Refusing update from remote peer c939f779969d413295046fa62887897e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 14:55:26.182253 14188 tablet.cc:2369] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:26.182617 14347 consensus_queue.cc:1035] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f95e816707a432a8d9d266bcaf5ad05" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:26.182771 14345 consensus_queue.cc:1035] T fbf4eaa77d9a4d9394349d4f27245748 P c939f779969d413295046fa62887897e [LEADER]: Connected to new peer: Peer: permanent_uuid: "61dec72e8c1b4a7aa06daa20f27bbc59" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:26.185508 14351 mvcc.cc:205] Tried to move back new op lower bound from 7333265924838535168 to 7333265924802473984. Current Snapshot: MvccSnapshot[applied={T|T < 7333265924838535168}]
I20260925 14:55:26.186852 14350 mvcc.cc:205] Tried to move back new op lower bound from 7333265924838535168 to 7333265924802473984. Current Snapshot: MvccSnapshot[applied={T|T < 7333265924838535168 or (T in {7333265924838535168})}]
I20260925 14:55:26.187518 13863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:26.187624 13863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:26.188400 13863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12f6cc9329a7426e9d31cbe7c3821e83: Sending DeleteTablet for 3 replicas of tablet fbf4eaa77d9a4d9394349d4f27245748
I20260925 14:55:26.188737 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf4eaa77d9a4d9394349d4f27245748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:26 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:39806
I20260925 14:55:26.188786 14254 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf4eaa77d9a4d9394349d4f27245748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:26 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:45390
I20260925 14:55:26.188802 13988 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf4eaa77d9a4d9394349d4f27245748 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:26 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:41120
I20260925 14:55:26.189313 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13926
I20260925 14:55:26.189972 14361 tablet_replica.cc:330] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: stopping tablet replica
I20260925 14:55:26.190085 14361 raft_consensus.cc:2229] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:26.190201 14361 raft_consensus.cc:2258] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:26.190333 14359 tablet_replica.cc:330] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: stopping tablet replica
I20260925 14:55:26.190428 14359 raft_consensus.cc:2229] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:26.190534 14359 raft_consensus.cc:2258] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:26.191001 14359 ts_tablet_manager.cc:1905] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:26.192756 14361 ts_tablet_manager.cc:1905] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:26.192939 14359 ts_tablet_manager.cc:1918] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:26.193019 14359 log.cc:1196] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/fbf4eaa77d9a4d9394349d4f27245748
I20260925 14:55:26.193332 14359 ts_tablet_manager.cc:1939] T fbf4eaa77d9a4d9394349d4f27245748 P 9f95e816707a432a8d9d266bcaf5ad05: Deleting consensus metadata
I20260925 14:55:26.193848 13851 catalog_manager.cc:4841] TS 9f95e816707a432a8d9d266bcaf5ad05 (127.9.141.66:38957): tablet fbf4eaa77d9a4d9394349d4f27245748 (table test-table [id=2b0647a76bfe49528bf50875b4fc9404]) successfully deleted
I20260925 14:55:26.194289 14361 ts_tablet_manager.cc:1918] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:26.194376 14361 log.cc:1196] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/fbf4eaa77d9a4d9394349d4f27245748
I20260925 14:55:26.194618 14361 ts_tablet_manager.cc:1939] T fbf4eaa77d9a4d9394349d4f27245748 P 61dec72e8c1b4a7aa06daa20f27bbc59: Deleting consensus metadata
I20260925 14:55:26.195111 13850 catalog_manager.cc:4841] TS 61dec72e8c1b4a7aa06daa20f27bbc59 (127.9.141.67:34627): tablet fbf4eaa77d9a4d9394349d4f27245748 (table test-table [id=2b0647a76bfe49528bf50875b4fc9404]) successfully deleted
W20260925 14:55:26.196606 13850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:46321 (error 108)
I20260925 14:55:26.196781 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14059
I20260925 14:55:26.198318 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.197443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46321 (local address 127.0.0.1:41128)
0925 14:55:26.198236 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.198288 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:46321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770}
W20260925 14:55:26.198454 13850 catalog_manager.cc:4584] TS c939f779969d413295046fa62887897e (127.9.141.65:46321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbf4eaa77d9a4d9394349d4f27245748: Network error: Client connection negotiation failed: client connection to 127.9.141.65:46321: connect: Connection refused (error 111)
I20260925 14:55:26.202122 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14192
I20260925 14:55:26.206300 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 13833
2026-09-25T14:55:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1936 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:26Z Disabled control of system clock
I20260925 14:55:26.233819 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:43299
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:43299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:55:26.312826 14366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.313040 14366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.313061 14366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.314170 14366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:26.314208 14366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.314224 14366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.314236 14366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:26.314248 14366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:26.315528 14366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:43299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:43299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:26.315727 14366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.315968 14366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.320703 14372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.320884 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.320940 14371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.321431 14366 server_base.cc:1034] running on GCE node
I20260925 14:55:26.321693 14366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.322024 14366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.323200 14366 hybrid_clock.cc:648] HybridClock initialized: now 1790348126323186 us; error 54 us; skew 500 ppm
I20260925 14:55:26.324544 14366 webserver.cc:460] Webserver started at http://127.9.141.126:37265/ using document root <none> and password file <none>
I20260925 14:55:26.324836 14366 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.324911 14366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.325059 14366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:26.326122 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "a52b1a6929d140a3982ed4ed04ed71ef"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "f22174a7d6032e8267f840338a59d401"
server_key_iv: "85d28c23459b52ed9e8fec344b467ff6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.326474 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "a52b1a6929d140a3982ed4ed04ed71ef"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "f22174a7d6032e8267f840338a59d401"
server_key_iv: "85d28c23459b52ed9e8fec344b467ff6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.327802 14366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:26.328650 14380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.328850 14366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:26.328958 14366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "a52b1a6929d140a3982ed4ed04ed71ef"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "f22174a7d6032e8267f840338a59d401"
server_key_iv: "85d28c23459b52ed9e8fec344b467ff6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.329031 14366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.332331 14366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.332630 14366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.332787 14366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.336841 14366 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:43299
I20260925 14:55:26.336892 14432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:43299 every 8 connection(s)
I20260925 14:55:26.337227 14366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:26.337841 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14366
I20260925 14:55:26.337942 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:26.338012 14433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.338140 9781 external_mini_cluster.cc:1428] Setting key d80b5e8dfc2904a84dd26a19a073fe2b
I20260925 14:55:26.341037 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.338894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.9.141.126:43299)
0925 14:55:26.339089 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.339098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.339130 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:55:26.339653 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.339663 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.339719 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.339879 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.340560 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.340563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.340563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.340568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.340620 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.340643 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.340750 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.340786 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.340841 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260925 14:55:26.341606 14433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef: Bootstrap starting.
I20260925 14:55:26.342222 14433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.342500 14433 log.cc:824] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.343323 14433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef: No bootstrap required, opened a new log
I20260925 14:55:26.344765 14433 raft_consensus.cc:348] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 43299 } }
I20260925 14:55:26.344898 14433 raft_consensus.cc:374] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.344920 14433 raft_consensus.cc:729] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52b1a6929d140a3982ed4ed04ed71ef, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.345039 14433 consensus_queue.cc:260] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 43299 } }
I20260925 14:55:26.345130 14433 raft_consensus.cc:388] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:26.345181 14433 raft_consensus.cc:482] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:26.345232 14433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.345780 14433 raft_consensus.cc:504] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 43299 } }
I20260925 14:55:26.345899 14433 leader_election.cc:302] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a52b1a6929d140a3982ed4ed04ed71ef; no voters:
I20260925 14:55:26.346073 14433 leader_election.cc:288] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:26.346141 14438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:26.346289 14438 raft_consensus.cc:686] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [term 1 LEADER]: Becoming Leader. State: Replica: a52b1a6929d140a3982ed4ed04ed71ef, State: Running, Role: LEADER
I20260925 14:55:26.346334 14433 sys_catalog.cc:564] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:26.346380 14438 consensus_queue.cc:237] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 43299 } }
I20260925 14:55:26.346724 14438 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader a52b1a6929d140a3982ed4ed04ed71ef. Latest consensus state: current_term: 1 leader_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 43299 } } }
I20260925 14:55:26.346810 14438 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:26.346736 14439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52b1a6929d140a3982ed4ed04ed71ef" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 43299 } } }
I20260925 14:55:26.346870 14439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:26.347079 14445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:26.347565 14445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:26.349282 14445 catalog_manager.cc:1347] Generated new cluster ID: 14304c78ea9045fcb7d743296fea68da
I20260925 14:55:26.349337 14445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:26.364240 14445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:26.364435 14445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:26.367142 14445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef: Generated new TSK 0
I20260925 14:55:26.367311 14445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:26.372440 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:43299
--builtin_ntp_servers=127.9.141.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:55:26.454424 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.454658 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.454694 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.455878 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.455941 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.455986 14457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:26.457432 14457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:43299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:26.457703 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.458000 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.463155 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.463163 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.463163 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.463460 14457 server_base.cc:1034] running on GCE node
I20260925 14:55:26.463652 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.463976 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.465147 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790348126465122 us; error 42 us; skew 500 ppm
I20260925 14:55:26.466495 14457 webserver.cc:460] Webserver started at http://127.9.141.65:39687/ using document root <none> and password file <none>
I20260925 14:55:26.466728 14457 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.466773 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.466939 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:26.468006 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "4fa7b0d96eee99dd26439a6db7aa1b9c"
server_key_iv: "50b6995911dafa5adc9382166051a4d9"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.468359 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "4fa7b0d96eee99dd26439a6db7aa1b9c"
server_key_iv: "50b6995911dafa5adc9382166051a4d9"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.469699 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:26.470526 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.470693 14457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:26.470778 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "4fa7b0d96eee99dd26439a6db7aa1b9c"
server_key_iv: "50b6995911dafa5adc9382166051a4d9"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.470841 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.474144 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.474427 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.474573 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.474792 14457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:26.475098 14457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:26.475153 14457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.475191 14457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:26.475219 14457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.480597 14457 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:39387
I20260925 14:55:26.480674 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:39387 every 8 connection(s)
I20260925 14:55:26.480986 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:26.483834 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.481703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:32907 (local address 127.9.141.126:43299)
0925 14:55:26.481751 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.481752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.481965 (+ 213us) server_negotiation.cc:407] Connection header received
0925 14:55:26.482405 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.482406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.482412 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.482477 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.483649 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.483650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.483650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.483654 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.483670 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.483673 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.483719 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.483772 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.483782 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:55:26.484045 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.481661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:43299 (local address 127.9.141.65:32907)
0925 14:55:26.481843 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.481882 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.482316 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.482547 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.482550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.482557 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.483506 (+ 949us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.483507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.483534 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.483561 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.483725 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.483727 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.483729 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.483778 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.483828 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 14:55:26.484570 14585 heartbeater.cc:347] Connected to a master server at 127.9.141.126:43299
I20260925 14:55:26.484668 14585 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:26.484845 14585 heartbeater.cc:511] Master 127.9.141.126:43299 requested a full tablet report, sending...
I20260925 14:55:26.485342 14397 ts_manager.cc:194] Registered new tserver with Master: c79c1c02aaf04cdd8ffd4fa9f8c81eac (127.9.141.65:39387)
I20260925 14:55:26.487061 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14457
I20260925 14:55:26.487228 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:26.487444 9781 external_mini_cluster.cc:1428] Setting key 658d9af344c4b3f70c69b0479d8031b6
I20260925 14:55:26.488336 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:43299
--builtin_ntp_servers=127.9.141.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:55:26.568822 14588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.569013 14588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.569034 14588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.570216 14588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.570259 14588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.570284 14588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:26.571621 14588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:43299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:26.571873 14588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.572165 14588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.577646 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.577646 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.577692 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.577960 14588 server_base.cc:1034] running on GCE node
I20260925 14:55:26.578190 14588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.578430 14588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.579597 14588 hybrid_clock.cc:648] HybridClock initialized: now 1790348126579591 us; error 37 us; skew 500 ppm
I20260925 14:55:26.580863 14588 webserver.cc:460] Webserver started at http://127.9.141.66:36585/ using document root <none> and password file <none>
I20260925 14:55:26.581126 14588 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.581205 14588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.581354 14588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:26.582413 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "1dcbfb047a82485aae152ae604055e57"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "ab5fd187f6a5684f11f9062ab3c51560"
server_key_iv: "fa48b2ce13cb7deb2b732b62c968b6c6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.582779 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "1dcbfb047a82485aae152ae604055e57"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "ab5fd187f6a5684f11f9062ab3c51560"
server_key_iv: "fa48b2ce13cb7deb2b732b62c968b6c6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.584110 14588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:26.584936 14602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.585111 14588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.585209 14588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "1dcbfb047a82485aae152ae604055e57"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "ab5fd187f6a5684f11f9062ab3c51560"
server_key_iv: "fa48b2ce13cb7deb2b732b62c968b6c6"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.585294 14588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.588676 14588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.588950 14588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.589097 14588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.589293 14588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:26.589601 14588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:26.589655 14588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.589694 14588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:26.589721 14588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.595201 14588 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:33199
I20260925 14:55:26.595292 14715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:33199 every 8 connection(s)
I20260925 14:55:26.595587 14588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:26.598495 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.596319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:44925 (local address 127.9.141.126:43299)
0925 14:55:26.596403 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.596406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.596666 (+ 260us) server_negotiation.cc:407] Connection header received
0925 14:55:26.597095 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.597096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.597101 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.597153 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.598285 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.598286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.598286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.598290 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.598303 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.598307 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.598355 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.598418 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.598430 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:55:26.598651 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.596319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:43299 (local address 127.9.141.66:44925)
0925 14:55:26.596549 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.596581 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.597014 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.597203 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.597207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.597214 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.598142 (+ 928us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.598142 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.598170 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.598197 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.598355 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.598357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.598359 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.598411 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.598456 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":113,"threads_started":1}
I20260925 14:55:26.599144 14716 heartbeater.cc:347] Connected to a master server at 127.9.141.126:43299
I20260925 14:55:26.599249 14716 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:26.599395 14716 heartbeater.cc:511] Master 127.9.141.126:43299 requested a full tablet report, sending...
I20260925 14:55:26.599751 14397 ts_manager.cc:194] Registered new tserver with Master: 1dcbfb047a82485aae152ae604055e57 (127.9.141.66:33199)
I20260925 14:55:26.602942 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14588
I20260925 14:55:26.603060 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:26.603222 9781 external_mini_cluster.cc:1428] Setting key 8175fbaddc8f42653bd32c0099ef3f4a
I20260925 14:55:26.604125 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:43299
--builtin_ntp_servers=127.9.141.84:33393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:55:26.682996 14719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.683195 14719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.683221 14719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.684432 14719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.684469 14719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.684496 14719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:26.685871 14719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:33393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:43299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:26.686111 14719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.686395 14719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.691550 14727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.691591 14719 server_base.cc:1034] running on GCE node
W20260925 14:55:26.691560 14725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.691558 14724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.691957 14719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.692198 14719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.693332 14719 hybrid_clock.cc:648] HybridClock initialized: now 1790348126693289 us; error 55 us; skew 500 ppm
I20260925 14:55:26.694546 14719 webserver.cc:460] Webserver started at http://127.9.141.67:35457/ using document root <none> and password file <none>
I20260925 14:55:26.694782 14719 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.694826 14719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.694988 14719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:26.696043 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "d4e33579eebd4cb3b52ebd15d8dc272c"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "415a145c1cc8dca2fca72485f58dc068"
server_key_iv: "04f94cf62d745cafd99793a53c4aec55"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.696472 14719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "d4e33579eebd4cb3b52ebd15d8dc272c"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "415a145c1cc8dca2fca72485f58dc068"
server_key_iv: "04f94cf62d745cafd99793a53c4aec55"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.697813 14719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:26.698727 14733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.698945 14719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:26.699065 14719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "d4e33579eebd4cb3b52ebd15d8dc272c"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "415a145c1cc8dca2fca72485f58dc068"
server_key_iv: "04f94cf62d745cafd99793a53c4aec55"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.699146 14719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.702558 14719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.702770 14719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.702862 14719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.703016 14719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:26.703313 14719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:26.703353 14719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.703373 14719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:26.703418 14719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.709151 14719 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:36237
I20260925 14:55:26.709223 14846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:36237 every 8 connection(s)
I20260925 14:55:26.709546 14719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:26.716320 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.710353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:46857 (local address 127.9.141.126:43299)
0925 14:55:26.710409 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.710411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.710630 (+ 219us) server_negotiation.cc:407] Connection header received
0925 14:55:26.711072 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.711073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.711078 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.711145 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.716085 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.716087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.716087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.716091 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.716106 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.716110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.716166 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.716254 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.716266 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:55:26.716583 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.710332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:43299 (local address 127.9.141.67:46857)
0925 14:55:26.710502 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.710540 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.710990 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.711198 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.711202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.711209 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.715903 (+ 4694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.715904 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.715936 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.715976 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.716181 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.716185 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.716188 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.716250 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.716302 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 14:55:26.717111 14847 heartbeater.cc:347] Connected to a master server at 127.9.141.126:43299
I20260925 14:55:26.717221 14847 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:26.717363 14847 heartbeater.cc:511] Master 127.9.141.126:43299 requested a full tablet report, sending...
I20260925 14:55:26.717795 14397 ts_manager.cc:194] Registered new tserver with Master: d4e33579eebd4cb3b52ebd15d8dc272c (127.9.141.67:36237)
I20260925 14:55:26.718628 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14719
I20260925 14:55:26.718707 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:26.718837 9781 external_mini_cluster.cc:1428] Setting key 6b703e7636e2f688d68d0eafdfa7ea42
I20260925 14:55:26.719934 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:26.722518 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.721476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.9.141.126:43299)
0925 14:55:26.721532 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.721534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.721645 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:55:26.721685 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.721686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.721690 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.721748 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.722353 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.722355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.722355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.722358 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.722370 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.722373 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.722423 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.722458 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.722468 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:55:26.723955 14397 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:26.725592 14397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:26.729238 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.727690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36237 (local address 127.0.0.1:43298)
0925 14:55:26.727860 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.727898 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.727952 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.728177 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.728183 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.728189 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.728780 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.728781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.728796 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.728825 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.729074 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.729080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.729083 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.729146 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.729159 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260925 14:55:26.729516 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.727859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.9.141.67:36237)
0925 14:55:26.728027 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.728032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.728043 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:26.728071 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.728073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.728084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.728169 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.728903 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.728910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.728911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.728915 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.728951 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.728974 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.729048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.729421 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.729439 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260925 14:55:26.729516 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.727690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39387 (local address 127.0.0.1:41652)
0925 14:55:26.727860 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.727898 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.727952 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.728677 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.728678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.728680 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.729133 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.729134 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.729140 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.729151 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.729385 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.729386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.729387 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.729433 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.729444 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":85,"threads_started":1}
I20260925 14:55:26.729940 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.728036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.9.141.65:39387)
0925 14:55:26.728226 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.728230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.728240 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:26.728266 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.728269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.728278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.728354 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.729228 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.729234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.729234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.729239 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.729274 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.729301 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.729359 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.729816 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.729835 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260925 14:55:26.730086 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.727712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:33199 (local address 127.0.0.1:59450)
0925 14:55:26.728037 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.728046 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.728086 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.728966 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.728967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.728969 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.729698 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.729698 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.729703 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.729711 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.729936 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.729938 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.729938 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.730005 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.730020 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":104,"threads_started":1}
I20260925 14:55:26.730279 14781 tablet_service.cc:1467] Processing CreateTablet for tablet 53b0fd9781e44348ab3191d11401399d (DEFAULT_TABLE table=test-table [id=e2edf1b36d56407d85b60f4ba8ffed6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:26.730512 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 53b0fd9781e44348ab3191d11401399d (DEFAULT_TABLE table=test-table [id=e2edf1b36d56407d85b60f4ba8ffed6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:26.730738 14781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b0fd9781e44348ab3191d11401399d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.730849 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b0fd9781e44348ab3191d11401399d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.732688 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.728229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.9.141.66:33199)
0925 14:55:26.728823 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.728828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.728840 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:26.728866 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.728868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.728876 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.728960 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.729786 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.729795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.729796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.729801 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.729834 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.729859 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.729920 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.732571 (+ 2651us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.732588 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":461,"threads_started":1}
I20260925 14:55:26.733386 14650 tablet_service.cc:1467] Processing CreateTablet for tablet 53b0fd9781e44348ab3191d11401399d (DEFAULT_TABLE table=test-table [id=e2edf1b36d56407d85b60f4ba8ffed6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:26.733702 14650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53b0fd9781e44348ab3191d11401399d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.733876 14861 tablet_bootstrap.cc:492] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac: Bootstrap starting.
I20260925 14:55:26.734421 14861 tablet_bootstrap.cc:654] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.734714 14861 log.cc:824] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.735438 14861 tablet_bootstrap.cc:492] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac: No bootstrap required, opened a new log
I20260925 14:55:26.735527 14861 ts_tablet_manager.cc:1397] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:26.736759 14863 tablet_bootstrap.cc:492] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Bootstrap starting.
I20260925 14:55:26.737556 14863 tablet_bootstrap.cc:654] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.737635 14864 tablet_bootstrap.cc:492] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Bootstrap starting.
I20260925 14:55:26.737833 14861 raft_consensus.cc:348] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.738010 14861 raft_consensus.cc:374] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.738059 14861 raft_consensus.cc:729] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79c1c02aaf04cdd8ffd4fa9f8c81eac, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.737986 14863 log.cc:824] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.738197 14861 consensus_queue.cc:260] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.738332 14864 tablet_bootstrap.cc:654] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.738473 14861 ts_tablet_manager.cc:1428] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:26.738518 14585 heartbeater.cc:503] Master 127.9.141.126:43299 was elected leader, sending a full tablet report...
I20260925 14:55:26.738655 14864 log.cc:824] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.739053 14863 tablet_bootstrap.cc:492] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: No bootstrap required, opened a new log
I20260925 14:55:26.739106 14863 ts_tablet_manager.cc:1397] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:26.739986 14864 tablet_bootstrap.cc:492] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: No bootstrap required, opened a new log
I20260925 14:55:26.740042 14864 ts_tablet_manager.cc:1397] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:26.741219 14863 raft_consensus.cc:348] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.741361 14863 raft_consensus.cc:374] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.741405 14863 raft_consensus.cc:729] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcbfb047a82485aae152ae604055e57, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.741533 14863 consensus_queue.cc:260] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.741773 14863 ts_tablet_manager.cc:1428] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:26.741850 14716 heartbeater.cc:503] Master 127.9.141.126:43299 was elected leader, sending a full tablet report...
I20260925 14:55:26.742331 14864 raft_consensus.cc:348] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.742501 14864 raft_consensus.cc:374] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.742545 14864 raft_consensus.cc:729] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e33579eebd4cb3b52ebd15d8dc272c, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.742660 14864 consensus_queue.cc:260] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.742910 14864 ts_tablet_manager.cc:1428] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:55:26.742983 14847 heartbeater.cc:503] Master 127.9.141.126:43299 was elected leader, sending a full tablet report...
I20260925 14:55:26.750054 14868 raft_consensus.cc:482] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:26.750131 14868 raft_consensus.cc:504] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.750448 14868 leader_election.cc:288] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c79c1c02aaf04cdd8ffd4fa9f8c81eac (127.9.141.65:39387), d4e33579eebd4cb3b52ebd15d8dc272c (127.9.141.67:36237)
I20260925 14:55:26.751962 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.750623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39387 (local address 127.9.141.66:35863)
0925 14:55:26.750677 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.750690 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.750733 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.751017 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.751019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.751022 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.751549 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.751550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.751558 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.751571 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.751765 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.751766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.751767 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.751865 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.751880 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:55:26.751986 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.750839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:35863 (local address 127.9.141.65:39387)
0925 14:55:26.750901 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.750902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.750911 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:26.750933 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.750934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.750940 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.750996 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.751672 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.751674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.751674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.751678 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.751695 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.751700 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.751757 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.751879 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.751892 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:55:26.752081 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.750705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:36013 (local address 127.9.141.67:36237)
0925 14:55:26.750758 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.750760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.750932 (+ 172us) server_negotiation.cc:407] Connection header received
0925 14:55:26.750961 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.750962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.750966 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.751041 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.751842 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.751843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.751844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.751849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.751866 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.751871 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.751931 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.751989 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.752003 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:55:26.752296 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.750624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36237 (local address 127.9.141.66:36013)
0925 14:55:26.750835 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.750844 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.750917 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.751064 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.751065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.751067 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:26.751743 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:26.751744 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:26.751750 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:26.751762 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:26.751936 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:26.751937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:26.751938 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:55:26.751988 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:55:26.752000 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":64,"thread_start_us":86,"threads_started":1}
I20260925 14:55:26.752373 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b0fd9781e44348ab3191d11401399d" candidate_uuid: "1dcbfb047a82485aae152ae604055e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" is_pre_election: true
I20260925 14:55:26.752522 14539 raft_consensus.cc:2454] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcbfb047a82485aae152ae604055e57 in term 0.
I20260925 14:55:26.752749 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b0fd9781e44348ab3191d11401399d" candidate_uuid: "1dcbfb047a82485aae152ae604055e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" is_pre_election: true
I20260925 14:55:26.752871 14801 raft_consensus.cc:2454] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcbfb047a82485aae152ae604055e57 in term 0.
I20260925 14:55:26.753060 14603 leader_election.cc:302] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcbfb047a82485aae152ae604055e57, c79c1c02aaf04cdd8ffd4fa9f8c81eac; no voters:
I20260925 14:55:26.753189 14868 raft_consensus.cc:2781] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:26.753259 14868 raft_consensus.cc:482] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:26.753295 14868 raft_consensus.cc:3037] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.753983 14868 raft_consensus.cc:504] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.754137 14868 leader_election.cc:288] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [CANDIDATE]: Term 1 election: Requested vote from peers c79c1c02aaf04cdd8ffd4fa9f8c81eac (127.9.141.65:39387), d4e33579eebd4cb3b52ebd15d8dc272c (127.9.141.67:36237)
I20260925 14:55:26.754276 14801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b0fd9781e44348ab3191d11401399d" candidate_uuid: "1dcbfb047a82485aae152ae604055e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c"
I20260925 14:55:26.754352 14801 raft_consensus.cc:3037] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.754997 14801 raft_consensus.cc:2454] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcbfb047a82485aae152ae604055e57 in term 1.
I20260925 14:55:26.755128 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53b0fd9781e44348ab3191d11401399d" candidate_uuid: "1dcbfb047a82485aae152ae604055e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac"
I20260925 14:55:26.755204 14539 raft_consensus.cc:3037] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.755240 14606 leader_election.cc:302] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcbfb047a82485aae152ae604055e57, d4e33579eebd4cb3b52ebd15d8dc272c; no voters:
I20260925 14:55:26.755339 14868 raft_consensus.cc:2781] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:26.755491 14868 raft_consensus.cc:686] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 1 LEADER]: Becoming Leader. State: Replica: 1dcbfb047a82485aae152ae604055e57, State: Running, Role: LEADER
I20260925 14:55:26.755599 14868 consensus_queue.cc:237] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } }
I20260925 14:55:26.756042 14539 raft_consensus.cc:2454] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcbfb047a82485aae152ae604055e57 in term 1.
I20260925 14:55:26.756412 14397 catalog_manager.cc:5495] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dcbfb047a82485aae152ae604055e57 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "1dcbfb047a82485aae152ae604055e57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcbfb047a82485aae152ae604055e57" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 33199 } health_report { overall_health: HEALTHY } } }
I20260925 14:55:26.765679 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.764567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.9.141.66:33199)
0925 14:55:26.764629 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.764631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.764641 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:26.764729 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.764731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.764741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.764797 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.765457 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.765458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.765458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.765462 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.765476 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.765480 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.765533 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.765593 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.765607 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:55:26.767251 14801 raft_consensus.cc:1261] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 1 FOLLOWER]: Refusing update from remote peer 1dcbfb047a82485aae152ae604055e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:26.767268 14539 raft_consensus.cc:1261] T 53b0fd9781e44348ab3191d11401399d P c79c1c02aaf04cdd8ffd4fa9f8c81eac [term 1 FOLLOWER]: Refusing update from remote peer 1dcbfb047a82485aae152ae604055e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:26.767580 14871 consensus_queue.cc:1035] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e33579eebd4cb3b52ebd15d8dc272c" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:26.767709 14868 consensus_queue.cc:1035] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c79c1c02aaf04cdd8ffd4fa9f8c81eac" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:26.772038 14396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:26.772116 14396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:26.772847 14396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a52b1a6929d140a3982ed4ed04ed71ef: Sending DeleteTablet for 3 replicas of tablet 53b0fd9781e44348ab3191d11401399d
I20260925 14:55:26.773178 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b0fd9781e44348ab3191d11401399d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:26 UTC) from {username='slave'} at 127.0.0.1:41652
I20260925 14:55:26.773178 14781 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b0fd9781e44348ab3191d11401399d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:26 UTC) from {username='slave'} at 127.0.0.1:43298
I20260925 14:55:26.773393 14884 tablet_replica.cc:330] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: stopping tablet replica
I20260925 14:55:26.773495 14884 raft_consensus.cc:2229] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:26.773554 14650 tablet_service.cc:1514] Processing DeleteTablet for tablet 53b0fd9781e44348ab3191d11401399d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:26 UTC) from {username='slave'} at 127.0.0.1:59450
I20260925 14:55:26.773602 14884 raft_consensus.cc:2258] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:26.773903 14886 tablet_replica.cc:330] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: stopping tablet replica
I20260925 14:55:26.774076 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14457
I20260925 14:55:26.774096 14884 ts_tablet_manager.cc:1905] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:26.774143 14886 raft_consensus.cc:2229] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:26.774224 14886 raft_consensus.cc:2258] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:26.774674 14886 ts_tablet_manager.cc:1905] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:26.775686 14884 ts_tablet_manager.cc:1918] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:26.775746 14884 log.cc:1196] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/53b0fd9781e44348ab3191d11401399d
I20260925 14:55:26.775992 14884 ts_tablet_manager.cc:1939] T 53b0fd9781e44348ab3191d11401399d P d4e33579eebd4cb3b52ebd15d8dc272c: Deleting consensus metadata
I20260925 14:55:26.776472 14384 catalog_manager.cc:4841] TS d4e33579eebd4cb3b52ebd15d8dc272c (127.9.141.67:36237): tablet 53b0fd9781e44348ab3191d11401399d (table test-table [id=e2edf1b36d56407d85b60f4ba8ffed6b]) successfully deleted
I20260925 14:55:26.776592 14886 ts_tablet_manager.cc:1918] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:26.776636 14886 log.cc:1196] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/53b0fd9781e44348ab3191d11401399d
I20260925 14:55:26.776823 14886 ts_tablet_manager.cc:1939] T 53b0fd9781e44348ab3191d11401399d P 1dcbfb047a82485aae152ae604055e57: Deleting consensus metadata
I20260925 14:55:26.777117 14383 catalog_manager.cc:4841] TS 1dcbfb047a82485aae152ae604055e57 (127.9.141.66:33199): tablet 53b0fd9781e44348ab3191d11401399d (table test-table [id=e2edf1b36d56407d85b60f4ba8ffed6b]) successfully deleted
W20260925 14:55:26.782681 14381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:39387 (error 108)
I20260925 14:55:26.783222 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.783089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39387 (local address 127.0.0.1:41664)
0925 14:55:26.783147 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.783187 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:39387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:55:26.783268 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14588
W20260925 14:55:26.783422 14381 catalog_manager.cc:4584] TS c79c1c02aaf04cdd8ffd4fa9f8c81eac (127.9.141.65:39387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53b0fd9781e44348ab3191d11401399d: Network error: Client connection negotiation failed: client connection to 127.9.141.65:39387: connect: Connection refused (error 111)
I20260925 14:55:26.788017 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14719
I20260925 14:55:26.792601 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14366
2026-09-25T14:55:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (585 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:55:26.812219 9781 mini_oidc.cc:152] Starting JWKS server
I20260925 14:55:26.813508 9781 webserver.cc:460] Webserver started at https://127.0.0.1:41007/ using document root <none> and password file <none>
I20260925 14:55:26.813637 9781 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:55:26.813827 9781 webserver.cc:460] Webserver started at http://127.0.0.1:41343/ using document root <none> and password file <none>
2026-09-25T14:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:26Z Disabled control of system clock
I20260925 14:55:26.821779 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36211
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36211
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41007/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/testchainca.cert with env {}
W20260925 14:55:26.902863 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:26.903117 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:26.903155 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:26.904403 14893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:26.904444 14893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:26.904458 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:26.904471 14893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:26.904484 14893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:55:26.904496 14893 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:55:26.904546 14893 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41007/jwks.json
I20260925 14:55:26.906092 14893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41007/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:26.906340 14893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:26.906598 14893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:26.911826 14899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.911823 14898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:26.912014 14901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:26.912065 14893 server_base.cc:1034] running on GCE node
I20260925 14:55:26.912472 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:26.912782 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:26.913964 14893 hybrid_clock.cc:648] HybridClock initialized: now 1790348126913937 us; error 46 us; skew 500 ppm
I20260925 14:55:26.915288 14893 webserver.cc:460] Webserver started at http://127.9.141.126:34099/ using document root <none> and password file <none>
I20260925 14:55:26.915534 14893 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:26.915603 14893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:26.915743 14893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:26.916873 14893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "759bd58c4ef2475683cc32130121f78a"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "67077f0cc0482e2f5c959f474608b16b"
server_key_iv: "30632b15a4b97a82c3cdf1a4a90ec792"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.917246 14893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "759bd58c4ef2475683cc32130121f78a"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "67077f0cc0482e2f5c959f474608b16b"
server_key_iv: "30632b15a4b97a82c3cdf1a4a90ec792"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.918562 14893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:55:26.919332 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:26.919623 14893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:26.919752 14893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "759bd58c4ef2475683cc32130121f78a"
format_stamp: "Formatted at 2026-09-25 14:55:26 on dist-test-slave-1sv7"
server_key: "67077f0cc0482e2f5c959f474608b16b"
server_key_iv: "30632b15a4b97a82c3cdf1a4a90ec792"
server_key_version: "encryptionkey@0"
I20260925 14:55:26.919843 14893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:26.929793 14893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:26.930085 14893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:26.930236 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:26.934662 14893 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:36211
I20260925 14:55:26.934693 14959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:36211 every 8 connection(s)
I20260925 14:55:26.935048 14893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:26.936091 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:26.936995 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14893
I20260925 14:55:26.937179 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:26.937395 9781 external_mini_cluster.cc:1428] Setting key 4d2d5526ea62040576bfb56d6c229b41
I20260925 14:55:26.939445 14960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a: Bootstrap starting.
I20260925 14:55:26.940171 14960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:26.940544 14960 log.cc:824] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:26.941314 14960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a: No bootstrap required, opened a new log
I20260925 14:55:26.943123 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.937967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58372 (local address 127.9.141.126:36211)
0925 14:55:26.938170 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.938177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.938202 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:55:26.938601 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.938608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.938666 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.938778 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:26.939038 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.939992 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.941199 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.941656 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.941744 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.942425 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:26.942428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:26.942428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:26.942585 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:26.942623 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:26.942634 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:26.942693 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:26.942806 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:55:26.942863 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260925 14:55:26.943851 14960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759bd58c4ef2475683cc32130121f78a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36211 } }
I20260925 14:55:26.944061 14960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:26.944135 14960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759bd58c4ef2475683cc32130121f78a, State: Initialized, Role: FOLLOWER
I20260925 14:55:26.944270 14960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759bd58c4ef2475683cc32130121f78a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36211 } }
I20260925 14:55:26.944358 14960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:26.944420 14960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:26.944465 14960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:26.944466 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:55:26.945009 14960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759bd58c4ef2475683cc32130121f78a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36211 } }
I20260925 14:55:26.945139 14960 leader_election.cc:302] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759bd58c4ef2475683cc32130121f78a; no voters:
I20260925 14:55:26.945305 14960 leader_election.cc:288] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:26.945369 14965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:26.945518 14965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [term 1 LEADER]: Becoming Leader. State: Replica: 759bd58c4ef2475683cc32130121f78a, State: Running, Role: LEADER
I20260925 14:55:26.945626 14960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:26.945639 14965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759bd58c4ef2475683cc32130121f78a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36211 } }
I20260925 14:55:26.946005 14965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759bd58c4ef2475683cc32130121f78a. Latest consensus state: current_term: 1 leader_uuid: "759bd58c4ef2475683cc32130121f78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759bd58c4ef2475683cc32130121f78a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36211 } } }
I20260925 14:55:26.946099 14965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:26.946014 14967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759bd58c4ef2475683cc32130121f78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759bd58c4ef2475683cc32130121f78a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36211 } } }
I20260925 14:55:26.946249 14967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:26.946429 14970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:26.946949 14970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:26.948642 14970 catalog_manager.cc:1347] Generated new cluster ID: 0fd77d80de564e2792c8306e849ef9d5
I20260925 14:55:26.948702 14970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:26.959862 14970 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:26.960363 14970 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:26.965690 14970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759bd58c4ef2475683cc32130121f78a: Generated new TSK 0
I20260925 14:55:26.965884 14970 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:55:26.991851 14990 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:55:26.991981 14990 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 14:55:26.992667 14989 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:26.975853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36211 (local address 127.0.0.1:58388)
0925 14:55:26.976057 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:26.976067 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:26.976128 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:26.976303 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:26.976304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:26.976306 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:55:26.976356 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.976358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.977520 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.977521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.978222 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:26.978225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.978763 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.978764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:26.978818 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.978837 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:55:26.992144 (+ 13307us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:55:26.992257 (+ 113us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.141.126:36211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.141.126:36211: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 14:55:26.992992 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:26.975933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.9.141.126:36211)
0925 14:55:26.975996 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:26.975999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:26.976127 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:55:26.976201 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:26.976202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:26.976215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:26.976303 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:55:26.976463 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.977346 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.978315 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:26.978666 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:26.978745 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:26.978960 (+ 215us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:55:26.991955 (+ 12995us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:55:26.992912 (+ 957us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:55:26.993034 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14893
W20260925 14:55:26.993054 14963 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58388: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41007/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-09-25T14:55:27Z chronyd exiting
I20260925 14:55:27.029006 9781 test_util.cc:175] -----------------------------------------------
I20260925 14:55:27.029095 9781 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348105363960-9781-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:55:27.029858 9781 mini_oidc.cc:152] Starting JWKS server
I20260925 14:55:27.030491 9781 webserver.cc:460] Webserver started at https://127.0.0.1:35627/ using document root <none> and password file <none>
I20260925 14:55:27.030530 9781 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:55:27.030678 9781 webserver.cc:460] Webserver started at http://127.0.0.1:46883/ using document root <none> and password file <none>
2026-09-25T14:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:27Z Disabled control of system clock
I20260925 14:55:27.038300 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44267
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:44267
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35627/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/testchainca.cert with env {}
W20260925 14:55:27.117290 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:27.117527 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:27.117563 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:27.118728 14996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:27.118796 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:27.118818 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:27.118845 14996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:27.118876 14996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:55:27.118901 14996 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:55:27.118927 14996 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35627/jwks.json
I20260925 14:55:27.120455 14996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:44267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35627/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:27.120726 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:27.120939 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:27.125825 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:27.125813 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:27.125813 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:27.126148 14996 server_base.cc:1034] running on GCE node
I20260925 14:55:27.126379 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:27.126614 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:27.127756 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790348127127731 us; error 41 us; skew 500 ppm
I20260925 14:55:27.129029 14996 webserver.cc:460] Webserver started at http://127.9.141.126:42147/ using document root <none> and password file <none>
I20260925 14:55:27.129267 14996 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:27.129311 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:27.129395 14996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:27.130398 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "344e9bea15764397b90012c76526efc6"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "7761f81fd08f885a8235a3903e07ea43"
server_key_iv: "5eee4750330412ed71fbb53646fa0656"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.130688 14996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "344e9bea15764397b90012c76526efc6"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "7761f81fd08f885a8235a3903e07ea43"
server_key_iv: "5eee4750330412ed71fbb53646fa0656"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.131947 14996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:27.132746 15010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:27.132939 14996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:27.133021 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "344e9bea15764397b90012c76526efc6"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "7761f81fd08f885a8235a3903e07ea43"
server_key_iv: "5eee4750330412ed71fbb53646fa0656"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.133074 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:27.162451 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:27.162767 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:27.162873 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:27.166544 14996 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:44267
I20260925 14:55:27.166620 15062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:44267 every 8 connection(s)
I20260925 14:55:27.166945 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:27.167943 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:27.170176 15063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6: Bootstrap starting.
I20260925 14:55:27.170758 15063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:27.171044 15063 log.cc:824] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:27.171718 15063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6: No bootstrap required, opened a new log
I20260925 14:55:27.172580 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 14996
I20260925 14:55:27.172933 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:27.173223 9781 external_mini_cluster.cc:1428] Setting key 5d4bd235faa5a270a81f89ba142dc069
I20260925 14:55:27.173666 15063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e9bea15764397b90012c76526efc6" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44267 } }
I20260925 14:55:27.173878 15063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:27.173929 15063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 344e9bea15764397b90012c76526efc6, State: Initialized, Role: FOLLOWER
I20260925 14:55:27.174057 15063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e9bea15764397b90012c76526efc6" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44267 } }
I20260925 14:55:27.174162 15063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:27.174201 15063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:27.174264 15063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:27.174943 15063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e9bea15764397b90012c76526efc6" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44267 } }
I20260925 14:55:27.175067 15063 leader_election.cc:302] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 344e9bea15764397b90012c76526efc6; no voters:
I20260925 14:55:27.175238 15063 leader_election.cc:288] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:27.175297 15068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:27.175433 15068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [term 1 LEADER]: Becoming Leader. State: Replica: 344e9bea15764397b90012c76526efc6, State: Running, Role: LEADER
I20260925 14:55:27.175501 15063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:27.175527 15068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e9bea15764397b90012c76526efc6" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44267 } }
I20260925 14:55:27.175904 15069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "344e9bea15764397b90012c76526efc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e9bea15764397b90012c76526efc6" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44267 } } }
I20260925 14:55:27.175998 15069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:27.175892 15068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 344e9bea15764397b90012c76526efc6. Latest consensus state: current_term: 1 leader_uuid: "344e9bea15764397b90012c76526efc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "344e9bea15764397b90012c76526efc6" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44267 } } }
I20260925 14:55:27.176172 15068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:27.176273 15075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:27.176833 15075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:27.178507 15075 catalog_manager.cc:1347] Generated new cluster ID: 21ffa2c47489472ea980629abae72837
I20260925 14:55:27.178572 15075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:27.180209 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.173764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.9.141.126:44267)
0925 14:55:27.174028 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.174046 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.174082 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:55:27.174197 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.174205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.174298 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.174439 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:27.174667 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.176521 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.177717 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.178403 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.178567 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.179359 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:27.179363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:27.179363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:27.179565 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:27.179607 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:27.179622 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:27.179690 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:27.179809 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 14:55:27.179977 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260925 14:55:27.181200 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:55:27.186785 15075 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:27.187497 15075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:27.190567 15075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 344e9bea15764397b90012c76526efc6: Generated new TSK 0
I20260925 14:55:27.190783 15075 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:55:27.220572 15093 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:55:27.220665 15093 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:55:27.220963 15067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.205075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.9.141.126:44267)
0925 14:55:27.205119 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.205122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.205219 (+ 97us) server_negotiation.cc:407] Connection header received
0925 14:55:27.205287 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.205288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.205301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.205382 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:55:27.205565 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.206363 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.207153 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.207483 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.207557 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.207739 (+ 182us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:55:27.220745 (+ 13006us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:55:27.220891 (+ 146us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49060: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:55:27.221024 15067 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49060: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 14:55:27.221215 15092 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:27.204992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44267 (local address 127.0.0.1:49060)
0925 14:55:27.205146 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.205157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:27.205222 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:27.205408 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:27.205409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:27.205411 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:55:27.205462 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.205464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.206450 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.206451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.207069 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.207070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.207572 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.207573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.207634 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.207639 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:55:27.220856 (+ 13217us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:55:27.220956 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.141.126:44267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260925 14:55:27.221629 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 14996
2026-09-25T14:55:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:55:27.259132 9781 mini_oidc.cc:152] Starting JWKS server
I20260925 14:55:27.259819 9781 webserver.cc:460] Webserver started at https://127.0.0.1:37501/ using document root <none> and password file <none>
I20260925 14:55:27.259889 9781 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:55:27.260082 9781 webserver.cc:460] Webserver started at http://127.0.0.1:36453/ using document root <none> and password file <none>
2026-09-25T14:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:27Z Disabled control of system clock
I20260925 14:55:27.267532 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46595
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:46595
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:55:27.347018 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:27.347239 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:27.347261 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:27.348475 15099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:27.348516 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:27.348529 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:27.348541 15099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:27.348553 15099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:55:27.348567 15099 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:55:27.348577 15099 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37501/jwks.json
I20260925 14:55:27.349970 15099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:46595
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:27.350234 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:27.350535 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:27.355480 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:27.355487 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:27.355628 15099 server_base.cc:1034] running on GCE node
W20260925 14:55:27.355487 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:27.355957 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:27.356204 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:27.357367 15099 hybrid_clock.cc:648] HybridClock initialized: now 1790348127357339 us; error 60 us; skew 500 ppm
I20260925 14:55:27.358647 15099 webserver.cc:460] Webserver started at http://127.9.141.126:41223/ using document root <none> and password file <none>
I20260925 14:55:27.358913 15099 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:27.358985 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:27.359138 15099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:27.360209 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "c4506819d0e946b09d918bff97d89649"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "e46c5f3d01bf306dec12becd66c0b9d8"
server_key_iv: "7ed12afa4b417ac99e65cf3e786cdedb"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.360615 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "c4506819d0e946b09d918bff97d89649"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "e46c5f3d01bf306dec12becd66c0b9d8"
server_key_iv: "7ed12afa4b417ac99e65cf3e786cdedb"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.361939 15099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:27.362663 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:27.362902 15099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:27.363005 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "c4506819d0e946b09d918bff97d89649"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "e46c5f3d01bf306dec12becd66c0b9d8"
server_key_iv: "7ed12afa4b417ac99e65cf3e786cdedb"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.363080 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:27.390348 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:27.390695 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:27.390851 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:27.395004 15099 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:46595
I20260925 14:55:27.395040 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:46595 every 8 connection(s)
I20260925 14:55:27.395397 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:27.396065 15166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:27.398890 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649: Bootstrap starting.
I20260925 14:55:27.399488 15166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:27.399782 15166 log.cc:824] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:27.400545 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649: No bootstrap required, opened a new log
I20260925 14:55:27.401968 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15099
I20260925 14:55:27.402050 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:27.402233 9781 external_mini_cluster.cc:1428] Setting key ce4675172b951a47c63894e74cea93f2
I20260925 14:55:27.402168 15166 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4506819d0e946b09d918bff97d89649" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46595 } }
I20260925 14:55:27.402331 15166 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:27.402401 15166 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4506819d0e946b09d918bff97d89649, State: Initialized, Role: FOLLOWER
I20260925 14:55:27.402549 15166 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4506819d0e946b09d918bff97d89649" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46595 } }
I20260925 14:55:27.402638 15166 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:27.402678 15166 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:27.402721 15166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:27.403270 15166 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4506819d0e946b09d918bff97d89649" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46595 } }
I20260925 14:55:27.403391 15166 leader_election.cc:302] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4506819d0e946b09d918bff97d89649; no voters:
I20260925 14:55:27.403550 15166 leader_election.cc:288] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:27.403581 15171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:27.403738 15171 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [term 1 LEADER]: Becoming Leader. State: Replica: c4506819d0e946b09d918bff97d89649, State: Running, Role: LEADER
I20260925 14:55:27.403829 15171 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4506819d0e946b09d918bff97d89649" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46595 } }
I20260925 14:55:27.403877 15166 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:27.404234 15172 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4506819d0e946b09d918bff97d89649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4506819d0e946b09d918bff97d89649" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46595 } } }
I20260925 14:55:27.404310 15172 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:27.404743 15177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:27.404551 15173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4506819d0e946b09d918bff97d89649. Latest consensus state: current_term: 1 leader_uuid: "c4506819d0e946b09d918bff97d89649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4506819d0e946b09d918bff97d89649" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46595 } } }
I20260925 14:55:27.404954 15173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:27.405293 15177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:27.406788 15177 catalog_manager.cc:1347] Generated new cluster ID: 6460d9fa3f71451fbf92f3a1d9a46625
I20260925 14:55:27.406847 15177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:27.408708 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.402725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.9.141.126:46595)
0925 14:55:27.402973 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.402982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.403018 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:55:27.403135 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.403140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.403198 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.403336 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:27.403549 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.405461 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.406641 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.407181 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.407319 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.408021 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:27.408025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:27.408025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:27.408200 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:27.408233 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:27.408243 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:27.408322 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:27.408416 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:55:27.408495 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260925 14:55:27.409695 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:55:27.412189 15177 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:27.412914 15177 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:27.416121 15177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4506819d0e946b09d918bff97d89649: Generated new TSK 0
I20260925 14:55:27.416265 15177 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:55:27.425179 15195 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:27.421456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46595 (local address 127.0.0.1:48416)
0925 14:55:27.421595 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.421605 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:27.421664 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:27.421826 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:27.421827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:27.421829 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:55:27.421895 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.421897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.422595 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.422596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.423582 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.423585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.423938 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.423939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.424004 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.424009 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:55:27.424998 (+ 989us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:55:27.425087 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.141.126:46595: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260925 14:55:27.425177 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.421550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.9.141.126:46595)
0925 14:55:27.421589 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.421591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.421670 (+ 79us) server_negotiation.cc:407] Connection header received
0925 14:55:27.421729 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.421730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.421740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.421802 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:55:27.421962 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.422497 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.423670 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.423855 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.423910 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.424110 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:55:27.424889 (+ 779us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:55:27.425040 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:55:27.425236 15170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48416: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 14:55:27.425880 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15099
E20260925 14:55:27.436450 15196 webserver.cc:533] Webserver: sslize failed:
E20260925 14:55:27.436517 15196 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T14:55:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:27Z Disabled control of system clock
I20260925 14:55:27.472143 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:40717
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:40717
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/testchainca.cert with env {}
W20260925 14:55:27.554926 15200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:27.555122 15200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:27.555145 15200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:27.556417 15200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:27.556470 15200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:27.556483 15200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:27.556496 15200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:27.556540 15200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:55:27.556564 15200 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:55:27.556593 15200 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:55:27.558086 15200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:40717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:27.558338 15200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:27.558580 15200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:27.563532 15205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:27.563591 15206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:27.563606 15200 server_base.cc:1034] running on GCE node
W20260925 14:55:27.563591 15208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:27.564105 15200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:27.564463 15200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:27.565619 15200 hybrid_clock.cc:648] HybridClock initialized: now 1790348127565594 us; error 43 us; skew 500 ppm
I20260925 14:55:27.566884 15200 webserver.cc:460] Webserver started at http://127.9.141.126:44407/ using document root <none> and password file <none>
I20260925 14:55:27.567133 15200 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:27.567237 15200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:27.567411 15200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:27.568498 15200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "645fa2439ff74f14a79b4e62bddc5816"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "088cad78af7c3b1fd09aedceaf02092b"
server_key_iv: "2071db058d7efef17b5f6d2e8019dce8"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.568850 15200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "645fa2439ff74f14a79b4e62bddc5816"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "088cad78af7c3b1fd09aedceaf02092b"
server_key_iv: "2071db058d7efef17b5f6d2e8019dce8"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.570144 15200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:27.570926 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:27.571123 15200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:27.571208 15200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "645fa2439ff74f14a79b4e62bddc5816"
format_stamp: "Formatted at 2026-09-25 14:55:27 on dist-test-slave-1sv7"
server_key: "088cad78af7c3b1fd09aedceaf02092b"
server_key_iv: "2071db058d7efef17b5f6d2e8019dce8"
server_key_version: "encryptionkey@0"
I20260925 14:55:27.571254 15200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:27.578899 15200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:27.579124 15200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:27.579222 15200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:27.583035 15200 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:40717
I20260925 14:55:27.583104 15266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:40717 every 8 connection(s)
I20260925 14:55:27.583434 15200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:27.584510 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:27.586663 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15200
I20260925 14:55:27.586755 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:27.586845 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816: Bootstrap starting.
I20260925 14:55:27.586993 9781 external_mini_cluster.cc:1428] Setting key 22a6875285561135fab0c7e485282301
I20260925 14:55:27.587559 15267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:27.587886 15267 log.cc:824] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:27.588613 15267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816: No bootstrap required, opened a new log
I20260925 14:55:27.590404 15267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645fa2439ff74f14a79b4e62bddc5816" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40717 } }
I20260925 14:55:27.590562 15267 raft_consensus.cc:374] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:27.590607 15267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 645fa2439ff74f14a79b4e62bddc5816, State: Initialized, Role: FOLLOWER
I20260925 14:55:27.590744 15267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645fa2439ff74f14a79b4e62bddc5816" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40717 } }
I20260925 14:55:27.590826 15267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:27.590867 15267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:27.590914 15267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:27.591462 15267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645fa2439ff74f14a79b4e62bddc5816" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40717 } }
I20260925 14:55:27.591583 15267 leader_election.cc:302] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 645fa2439ff74f14a79b4e62bddc5816; no voters:
I20260925 14:55:27.591780 15267 leader_election.cc:288] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:27.591809 15272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:27.591913 15272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [term 1 LEADER]: Becoming Leader. State: Replica: 645fa2439ff74f14a79b4e62bddc5816, State: Running, Role: LEADER
I20260925 14:55:27.591993 15272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645fa2439ff74f14a79b4e62bddc5816" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40717 } }
I20260925 14:55:27.592037 15267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:27.592339 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "645fa2439ff74f14a79b4e62bddc5816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645fa2439ff74f14a79b4e62bddc5816" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40717 } } }
I20260925 14:55:27.592461 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:27.592547 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.587506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.9.141.126:40717)
0925 14:55:27.587743 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.587757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.587779 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:55:27.588242 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.588249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.588306 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.588494 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:27.588695 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.589563 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.590346 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.590866 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.591005 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.591695 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:27.591699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:27.591699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:27.591922 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:27.591968 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:27.591984 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:27.592049 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:27.592197 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:55:27.592283 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260925 14:55:27.592617 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 645fa2439ff74f14a79b4e62bddc5816. Latest consensus state: current_term: 1 leader_uuid: "645fa2439ff74f14a79b4e62bddc5816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "645fa2439ff74f14a79b4e62bddc5816" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40717 } } }
I20260925 14:55:27.592684 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:27.592864 15278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:27.593357 15278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:27.593374 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:55:27.595896 15278 catalog_manager.cc:1347] Generated new cluster ID: 6cfe27d6d6be48f2a65dbcec67efc65a
I20260925 14:55:27.595952 15278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:27.606905 15278 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:27.607638 15278 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:27.610934 15278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 645fa2439ff74f14a79b4e62bddc5816: Generated new TSK 0
I20260925 14:55:27.611177 15278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:28.093824 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:27.619934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.9.141.126:40717)
0925 14:55:27.619998 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:27.620000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:27.620091 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:55:27.620193 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:27.620194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:27.620206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:27.620282 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:55:27.620537 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.621038 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.621841 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.622042 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.622115 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.622309 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:55:28.093480 (+471171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:55:28.093733 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46326: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 14:55:28.093905 15270 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46326: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 14:55:28.094039 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:27.619854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40717 (local address 127.0.0.1:46326)
0925 14:55:27.620004 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:27.620015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:27.620105 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:27.620353 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:27.620354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:27.620355 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:55:27.620460 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.620466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.621137 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.621138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.621751 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:27.621753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:27.622132 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:27.622134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:27.622188 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:27.622194 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:55:28.093688 (+471494us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:55:28.093871 (+ 183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.141.126:40717: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260925 14:55:28.094408 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15200
2026-09-25T14:55:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (648 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:28 dist-test-slave-1sv7 krb5kdc[15308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:28 dist-test-slave-1sv7 krb5kdc[15308](info): set up 2 sockets
Sep 25 14:55:28 dist-test-slave-1sv7 krb5kdc[15308](info): commencing operation
krb5kdc: starting...
W20260925 14:55:29.328716 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:29.447774 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:42025
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:42025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:29.531155 15321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:29.531435 15321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:29.531476 15321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:29.533907 15321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:29.533996 15321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:29.534024 15321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:29.534051 15321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:29.534072 15321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:29.536969 15321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:42025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:42025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:29.537305 15321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:29.537626 15321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:29.543481 15329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:29.543476 15327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:29.543493 15326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:29.543730 15321 server_base.cc:1034] running on GCE node
I20260925 14:55:29.544014 15321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:29.544315 15321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:29.545545 15321 hybrid_clock.cc:648] HybridClock initialized: now 1790348129545515 us; error 42 us; skew 500 ppm
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:29.548029 15321 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:29.548497 15321 webserver.cc:460] Webserver started at http://127.9.141.126:39941/ using document root <none> and password file <none>
I20260925 14:55:29.548681 15321 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:29.548722 15321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:29.548817 15321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:29.549835 15321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "3d86071e2b9d4911bb7bed87603e5751"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "377e0e4fdc7318472bf22fa77f4ab080"
server_key_iv: "8ae69612bd610cea3682b8fe66781b53"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.550166 15321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "3d86071e2b9d4911bb7bed87603e5751"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "377e0e4fdc7318472bf22fa77f4ab080"
server_key_iv: "8ae69612bd610cea3682b8fe66781b53"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.551456 15321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:29.552206 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.552366 15321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:29.552479 15321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "3d86071e2b9d4911bb7bed87603e5751"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "377e0e4fdc7318472bf22fa77f4ab080"
server_key_iv: "8ae69612bd610cea3682b8fe66781b53"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.552552 15321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:29.570261 15321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:29.572481 15321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:29.572636 15321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:29.576282 15321 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:42025
I20260925 14:55:29.576359 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:42025 every 8 connection(s)
I20260925 14:55:29.576689 15321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:29.577620 15389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:29.580031 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751: Bootstrap starting.
I20260925 14:55:29.580672 15389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:29.580924 15389 log.cc:824] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:29.581646 15389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751: No bootstrap required, opened a new log
I20260925 14:55:29.582475 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15321
I20260925 14:55:29.582582 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:29.582804 9781 external_mini_cluster.cc:1428] Setting key 1d542465f659326d01d8058d55609aaa
I20260925 14:55:29.584538 15389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d86071e2b9d4911bb7bed87603e5751" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42025 } }
I20260925 14:55:29.584748 15389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:29.584791 15389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d86071e2b9d4911bb7bed87603e5751, State: Initialized, Role: FOLLOWER
I20260925 14:55:29.584944 15389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d86071e2b9d4911bb7bed87603e5751" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42025 } }
I20260925 14:55:29.585044 15389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:29.585083 15389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:29.585141 15389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:29.585868 15389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d86071e2b9d4911bb7bed87603e5751" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42025 } }
I20260925 14:55:29.586012 15389 leader_election.cc:302] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d86071e2b9d4911bb7bed87603e5751; no voters:
I20260925 14:55:29.586236 15389 leader_election.cc:288] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:29.586269 15394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:29.586431 15394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [term 1 LEADER]: Becoming Leader. State: Replica: 3d86071e2b9d4911bb7bed87603e5751, State: Running, Role: LEADER
I20260925 14:55:29.586526 15394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d86071e2b9d4911bb7bed87603e5751" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42025 } }
I20260925 14:55:29.586581 15389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:29.586869 15396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d86071e2b9d4911bb7bed87603e5751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d86071e2b9d4911bb7bed87603e5751" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42025 } } }
I20260925 14:55:29.586884 15395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d86071e2b9d4911bb7bed87603e5751. Latest consensus state: current_term: 1 leader_uuid: "3d86071e2b9d4911bb7bed87603e5751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d86071e2b9d4911bb7bed87603e5751" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42025 } } }
I20260925 14:55:29.587041 15396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:29.587064 15395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:29.587970 15403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:29.588661 15403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:29.590461 15403 catalog_manager.cc:1347] Generated new cluster ID: 91fa905e22004e44bacaaa4b9fd059ae
I20260925 14:55:29.590516 15403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:29.595435 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.583341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.9.141.126:42025)
0925 14:55:29.583515 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:29.583522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:29.583547 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:55:29.584442 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:29.584451 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:29.584501 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:29.584615 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:29.585529 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.586766 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.587772 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.588476 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.588587 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:29.591572 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:29.591577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:29.591577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:29.591591 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:29.593671 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:29.594152 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:29.594155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:29.594155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:29.594260 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:29.594493 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:29.594493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:29.594494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:29.594803 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:29.594964 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:29.595095 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:55:29.595229 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260925 14:55:29.598991 15403 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:29.599716 15403 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:29.604490 15403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d86071e2b9d4911bb7bed87603e5751: Generated new TSK 0
I20260925 14:55:29.604660 15403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:29.631398 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:42025
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:29.711727 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:29.711925 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:29.711947 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:29.713145 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:29.713187 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:29.713214 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:29.714697 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:42025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:29.714932 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:29.715179 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:29.720242 15423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:29.720398 15421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:29.720268 15420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:29.720492 15415 server_base.cc:1034] running on GCE node
I20260925 14:55:29.720770 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:29.721035 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:29.722208 15415 hybrid_clock.cc:648] HybridClock initialized: now 1790348129722194 us; error 46 us; skew 500 ppm
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:29.724949 15415 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:29.725327 15415 webserver.cc:460] Webserver started at http://127.9.141.65:38949/ using document root <none> and password file <none>
I20260925 14:55:29.725525 15415 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:29.725564 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:29.725687 15415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:29.726754 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "697ef2a62a7e47939bf69b651002679c"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "2dfce05c7bfbc26d1d5f0934fcdaf878"
server_key_iv: "1521c3638b96d67dac8e59edc191df46"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.727101 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "697ef2a62a7e47939bf69b651002679c"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "2dfce05c7bfbc26d1d5f0934fcdaf878"
server_key_iv: "1521c3638b96d67dac8e59edc191df46"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.728458 15415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:29.729187 15430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.729420 15415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:29.729489 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "697ef2a62a7e47939bf69b651002679c"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "2dfce05c7bfbc26d1d5f0934fcdaf878"
server_key_iv: "1521c3638b96d67dac8e59edc191df46"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.729540 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:29.745604 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:29.747686 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:29.747813 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:29.748019 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:29.748282 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:29.748315 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.748338 15415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:29.748353 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.754695 15415 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:42661
I20260925 14:55:29.754741 15543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:42661 every 8 connection(s)
I20260925 14:55:29.755070 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:29.756117 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15415
I20260925 14:55:29.756299 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:29.756562 9781 external_mini_cluster.cc:1428] Setting key 07d6ca7651d1e8473775231ed6f0d252
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
I20260925 14:55:29.765285 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.755820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:49179 (local address 127.9.141.126:42025)
0925 14:55:29.755881 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:29.755883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:29.756049 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:55:29.757332 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:29.757333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:29.757348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:29.757440 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:29.758294 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.758999 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.760004 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.760228 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.760306 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:29.762471 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:29.762472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:29.762472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:29.762479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:29.763868 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:29.764283 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:29.764284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:29.764285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:29.764342 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:29.764640 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:29.764641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:29.764641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:29.764759 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:29.764836 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:29.765111 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 14:55:29.765194 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:55:29.765470 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.755761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:42025 (local address 127.9.141.65:49179)
0925 14:55:29.755925 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:29.755970 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:29.757179 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:29.757536 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:29.757541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:29.757926 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:29.758199 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:29.758205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.759098 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.759099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:29.759899 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:29.759901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.760312 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.760313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:29.760482 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:29.761030 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:29.761038 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:29.762355 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:29.763979 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:29.763985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:29.763987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:29.764204 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:29.764456 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:29.764457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:29.764457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:29.764564 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:29.764827 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:29.764833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:29.764982 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:55:29.765089 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:55:29.765240 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260925 14:55:29.766042 15544 heartbeater.cc:347] Connected to a master server at 127.9.141.126:42025
I20260925 14:55:29.766160 15544 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:29.766350 15544 heartbeater.cc:511] Master 127.9.141.126:42025 requested a full tablet report, sending...
I20260925 14:55:29.766913 15353 ts_manager.cc:194] Registered new tserver with Master: 697ef2a62a7e47939bf69b651002679c (127.9.141.65:42661)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
I20260925 14:55:29.768362 15353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:49179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:29.783512 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:42025
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:29.869161 15549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:29.869413 15549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:29.869451 15549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:29.870657 15549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:29.870733 15549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:29.870786 15549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:29.872265 15549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:42025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:29.872566 15549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:29.872808 15549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:29.877821 15557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:29.877856 15554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:29.877838 15555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:29.878001 15549 server_base.cc:1034] running on GCE node
I20260925 14:55:29.878300 15549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:29.878551 15549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:29.879722 15549 hybrid_clock.cc:648] HybridClock initialized: now 1790348129879706 us; error 27 us; skew 500 ppm
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:29.882285 15549 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:29.882753 15549 webserver.cc:460] Webserver started at http://127.9.141.66:37197/ using document root <none> and password file <none>
I20260925 14:55:29.882992 15549 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:29.883036 15549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:29.883175 15549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:29.884205 15549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "3887f2dddd7f422fa394d81c36b2bfab"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "de1a9bb4de073ad42ff45aa2421da234"
server_key_iv: "b1b0cb09d39c5f37febda1d15b1d1ecb"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.884608 15549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "3887f2dddd7f422fa394d81c36b2bfab"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "de1a9bb4de073ad42ff45aa2421da234"
server_key_iv: "b1b0cb09d39c5f37febda1d15b1d1ecb"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.885946 15549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:29.886785 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.886993 15549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:29.887069 15549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "3887f2dddd7f422fa394d81c36b2bfab"
format_stamp: "Formatted at 2026-09-25 14:55:29 on dist-test-slave-1sv7"
server_key: "de1a9bb4de073ad42ff45aa2421da234"
server_key_iv: "b1b0cb09d39c5f37febda1d15b1d1ecb"
server_key_version: "encryptionkey@0"
I20260925 14:55:29.887176 15549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:29.899169 15549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:29.901522 15549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:29.901690 15549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:29.901947 15549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:29.902263 15549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:29.902318 15549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.902352 15549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:29.902379 15549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:29.908288 15549 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:37617
I20260925 14:55:29.908475 15677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:37617 every 8 connection(s)
I20260925 14:55:29.908746 15549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:55:29 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348129, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:29.918200 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15549
I20260925 14:55:29.918318 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:29.918320 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.909489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:43899 (local address 127.9.141.126:42025)
0925 14:55:29.909726 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:29.909728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:29.909893 (+ 165us) server_negotiation.cc:407] Connection header received
0925 14:55:29.910862 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:29.910863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:29.910876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:29.910944 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:29.911587 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.912113 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.913175 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.913370 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.913445 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:29.915601 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:29.915602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:29.915603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:29.915611 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:29.916886 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:29.917301 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:29.917302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:29.917302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:29.917363 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:29.917622 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:29.917623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:29.917623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:29.917740 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:29.917818 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:29.918138 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:55:29.918209 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260925 14:55:29.918520 9781 external_mini_cluster.cc:1428] Setting key f430b19ef42d10fe05de70886837881e
I20260925 14:55:29.918525 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:29.909454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:42025 (local address 127.9.141.66:43899)
0925 14:55:29.909745 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:29.909793 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:29.910742 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:29.911016 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:29.911020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:29.911283 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:29.911475 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:29.911479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.912210 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.912211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:29.913067 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:29.913070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:29.913462 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:29.913463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:29.913599 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:29.914090 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:29.914099 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:29.915470 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:29.916984 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:29.916991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:29.916993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:29.917219 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:29.917432 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:29.917433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:29.917434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:29.917546 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:29.917809 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:29.917815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:29.917953 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:55:29.918084 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 14:55:29.918262 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":95,"threads_started":1}
I20260925 14:55:29.919034 15678 heartbeater.cc:347] Connected to a master server at 127.9.141.126:42025
I20260925 14:55:29.919150 15678 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:29.919374 15678 heartbeater.cc:511] Master 127.9.141.126:42025 requested a full tablet report, sending...
I20260925 14:55:29.919817 15352 ts_manager.cc:194] Registered new tserver with Master: 3887f2dddd7f422fa394d81c36b2bfab (127.9.141.66:37617)
I20260925 14:55:29.920214 15352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:43899
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:29.942727 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:42025
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:30.022738 15683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:30.022971 15683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:30.023008 15683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:30.024145 15683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:30.024216 15683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:30.024260 15683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:30.025735 15683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:42025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:30.026011 15683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:30.026247 15683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:30.031561 15691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:30.031605 15689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:30.031648 15683 server_base.cc:1034] running on GCE node
W20260925 14:55:30.031628 15688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:30.032021 15683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:30.032263 15683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:30.033432 15683 hybrid_clock.cc:648] HybridClock initialized: now 1790348130033409 us; error 35 us; skew 500 ppm
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.035981 15683 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:30.036350 15683 webserver.cc:460] Webserver started at http://127.9.141.67:43747/ using document root <none> and password file <none>
I20260925 14:55:30.036587 15683 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:30.036631 15683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:30.036715 15683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:30.037766 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "6fec1f968ead436b96c2ff7227cfa424"
format_stamp: "Formatted at 2026-09-25 14:55:30 on dist-test-slave-1sv7"
server_key: "b4abbd4a7c90c17afeffd55e7285e8db"
server_key_iv: "deb8d19607068b6af5547fc0fb8701f1"
server_key_version: "encryptionkey@0"
I20260925 14:55:30.038069 15683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "6fec1f968ead436b96c2ff7227cfa424"
format_stamp: "Formatted at 2026-09-25 14:55:30 on dist-test-slave-1sv7"
server_key: "b4abbd4a7c90c17afeffd55e7285e8db"
server_key_iv: "deb8d19607068b6af5547fc0fb8701f1"
server_key_version: "encryptionkey@0"
I20260925 14:55:30.039335 15683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:30.039983 15698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:30.040170 15683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:30.040239 15683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "6fec1f968ead436b96c2ff7227cfa424"
format_stamp: "Formatted at 2026-09-25 14:55:30 on dist-test-slave-1sv7"
server_key: "b4abbd4a7c90c17afeffd55e7285e8db"
server_key_iv: "deb8d19607068b6af5547fc0fb8701f1"
server_key_version: "encryptionkey@0"
I20260925 14:55:30.040328 15683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:30.079952 15683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:30.082470 15683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:30.082594 15683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:30.082887 15683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:30.083232 15683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:30.083290 15683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:30.083333 15683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:30.083360 15683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:30.089470 15683 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:40081
I20260925 14:55:30.089524 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:40081 every 8 connection(s)
I20260925 14:55:30.089861 15683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:30.097798 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15683
I20260925 14:55:30.097863 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:30.098047 9781 external_mini_cluster.cc:1428] Setting key 9e81976056baeb50d4d5ff7458afc2f1
I20260925 14:55:30.099757 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.090700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:49831 (local address 127.9.141.126:42025)
0925 14:55:30.090750 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.090752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.090957 (+ 205us) server_negotiation.cc:407] Connection header received
0925 14:55:30.092032 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.092034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.092045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.092100 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.092791 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.093288 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.094371 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.094569 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.094630 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:30.096778 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:30.096780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:30.096780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:30.096788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:30.098391 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.098795 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.098796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.098797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.098858 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.099111 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.099112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.099113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.099231 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:30.099307 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:30.099587 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:55:30.099663 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:55:30.099951 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.090596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:42025 (local address 127.9.141.67:49831)
0925 14:55:30.090794 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:30.090832 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:30.091908 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:30.092169 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:30.092172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:30.092500 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:30.092697 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:30.092702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.093391 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.093393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:30.094261 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:30.094264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.094652 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.094653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:30.094757 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:30.095222 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:30.095231 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:30.096665 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:30.098465 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:30.098472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:30.098474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:30.098701 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:30.098924 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:30.098925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:30.098925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:30.099042 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:30.099298 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:30.099304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:30.099453 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:55:30.099567 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:55:30.099721 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260925 14:55:30.100489 15812 heartbeater.cc:347] Connected to a master server at 127.9.141.126:42025
I20260925 14:55:30.100571 15812 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:30.100767 15812 heartbeater.cc:511] Master 127.9.141.126:42025 requested a full tablet report, sending...
I20260925 14:55:30.101167 15352 ts_manager.cc:194] Registered new tserver with Master: 6fec1f968ead436b96c2ff7227cfa424 (127.9.141.67:40081)
I20260925 14:55:30.101662 15352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:49831
I20260925 14:55:30.109939 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/test-admin.keytab.
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.126364 9781 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:55:30.128139 15819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:30.127276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.9.141.65:42661)
0925 14:55:30.127518 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.127523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.127535 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:30.127560 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.127562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.127575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.127648 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.128030 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58670: BlockingRecv error: recv got EOF from 127.0.0.1:58670 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":114,"threads_started":1}
W20260925 14:55:30.128269 9781 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.129328 9781 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.130292 9781 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:30.138840 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.130967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.9.141.65:42661)
0925 14:55:30.131007 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.131009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.131064 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:55:30.131101 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.131102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.131111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.131170 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.131604 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.132512 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.133397 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.133732 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.133793 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:30.136111 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:30.136119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:30.136119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:30.136125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:30.137646 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.138021 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.138024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.138024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.138084 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.138266 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.138267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.138267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.138391 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:30.138502 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:30.138636 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:55:30.138700 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:55:30.140753 15819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:30.140229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.9.141.65:42661)
0925 14:55:30.140284 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.140286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.140300 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:55:30.140318 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.140318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.140326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.140371 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.140720 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58686: BlockingRecv error: recv got EOF from 127.0.0.1:58686 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 14:55:30.140846 9781 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.141853 9781 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.142966 9781 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:30.150583 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.143622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.9.141.65:42661)
0925 14:55:30.143703 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.143705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.143715 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:30.143750 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.143751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.143760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.143828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.144231 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.144817 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.145521 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.145733 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.145792 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:30.148127 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:30.148128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:30.148129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:30.148135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:30.149409 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.149738 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.149739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.149739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.149784 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.150002 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.150003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.150003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.150104 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:30.150173 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:30.150413 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:55:30.150512 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 14:55:30.152307 15819 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:30.151742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58702 (local address 127.9.141.65:42661)
0925 14:55:30.151809 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.151811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.151838 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:55:30.151908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.151909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.151918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.151962 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.152235 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58702: BlockingRecv error: recv got EOF from 127.0.0.1:58702 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 14:55:30.152483 9781 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348105363960-9781-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.153431 9781 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:30.154264 9781 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.65@KRBTEST.COM
I20260925 14:55:30.161520 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:30.154811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.9.141.65:42661)
0925 14:55:30.154889 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:30.154890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:30.154965 (+ 75us) server_negotiation.cc:407] Connection header received
0925 14:55:30.155027 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:30.155028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:30.155037 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:30.155119 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:30.155460 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.155919 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.156747 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:30.156946 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:30.157010 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:30.159162 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:30.159164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:30.159164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:30.159170 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:30.160342 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.160742 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.160743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.160743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.160793 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:30.161002 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:30.161003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:30.161003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:30.161109 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:30.161191 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:30.161310 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 14:55:30.161410 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:55:30.162228 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15415
I20260925 14:55:30.166724 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15549
I20260925 14:55:30.171258 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15683
I20260925 14:55:30.175515 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15321
2026-09-25T14:55:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1909 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15832](info): set up 2 sockets
Sep 25 14:55:30 dist-test-slave-1sv7 krb5kdc[15832](info): commencing operation
krb5kdc: starting...
W20260925 14:55:31.216732 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab.
Entry for principal oryx/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab.
I20260925 14:55:31.336213 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36855
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab
--principal=oryx/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:31.418557 15845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:31.418803 15845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:31.418828 15845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:31.420094 15845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:31.420135 15845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:31.420149 15845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:31.420163 15845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:31.420177 15845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:31.421800 15845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab
--principal=oryx/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:31.422013 15845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:31.422217 15845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:31.427253 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:31.427222 15851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:31.427235 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:31.427534 15845 server_base.cc:1034] running on GCE node
I20260925 14:55:31.427839 15845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:31.428164 15845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:31.429363 15845 hybrid_clock.cc:648] HybridClock initialized: now 1790348131429340 us; error 37 us; skew 500 ppm
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:31.431932 15845 init.cc:377] Logged in from keytab as oryx/127.9.141.126@KRBTEST.COM (short username oryx)
I20260925 14:55:31.432495 15845 webserver.cc:460] Webserver started at http://127.9.141.126:42549/ using document root <none> and password file <none>
I20260925 14:55:31.432747 15845 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:31.432814 15845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:31.432968 15845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:31.434088 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "a36adc204c1960e54e3f934dfd8910ca"
server_key_iv: "408960a6226844de4f204c47cd109f02"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.434460 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "a36adc204c1960e54e3f934dfd8910ca"
server_key_iv: "408960a6226844de4f204c47cd109f02"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.435830 15845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:31.436746 15860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.436998 15845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:31.437101 15845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "a36adc204c1960e54e3f934dfd8910ca"
server_key_iv: "408960a6226844de4f204c47cd109f02"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.437230 15845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:31.450493 15845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:31.453334 15845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:31.453501 15845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:31.457701 15845 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:36855
I20260925 14:55:31.457772 15912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:36855 every 8 connection(s)
I20260925 14:55:31.458134 15845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:31.459295 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:31.461057 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15845
I20260925 14:55:31.461153 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:31.461381 9781 external_mini_cluster.cc:1428] Setting key 8940f60a66334acf6415b967d7a33ae0
I20260925 14:55:31.462337 15913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Bootstrap starting.
I20260925 14:55:31.463073 15913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:31.463392 15913 log.cc:824] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:31.464167 15913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: No bootstrap required, opened a new log
I20260925 14:55:31.466363 15913 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } }
I20260925 14:55:31.466500 15913 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:31.466523 15913 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcbd9ff6bcbe4f428b3db34d35134b4c, State: Initialized, Role: FOLLOWER
I20260925 14:55:31.466661 15913 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } }
I20260925 14:55:31.466750 15913 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:31.466773 15913 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:31.466851 15913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:31.467470 15913 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } }
I20260925 14:55:31.467578 15913 leader_election.cc:302] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcbd9ff6bcbe4f428b3db34d35134b4c; no voters:
I20260925 14:55:31.467793 15913 leader_election.cc:288] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:31.467835 15918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:31.468048 15918 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 LEADER]: Becoming Leader. State: Replica: dcbd9ff6bcbe4f428b3db34d35134b4c, State: Running, Role: LEADER
I20260925 14:55:31.468205 15918 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } }
I20260925 14:55:31.468070 15913 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:31.468647 15920 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } }
I20260925 14:55:31.468794 15920 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:31.469014 15919 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbd9ff6bcbe4f428b3db34d35134b4c. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } }
I20260925 14:55:31.469203 15919 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:31.469882 15923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:31.470414 15923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:31.472991 15923 catalog_manager.cc:1347] Generated new cluster ID: 8daf7cc897dc4e01afca6554a3eb2ccf
I20260925 14:55:31.473043 15923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:31.474303 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.461988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.9.141.126:36855)
0925 14:55:31.462187 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:31.462197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:31.462232 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:55:31.462852 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:31.462862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:31.462927 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:31.463094 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:31.463794 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.465004 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.466153 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.466731 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.466879 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:31.470293 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:31.470297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:31.470298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:31.470310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:31.472480 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:31.472915 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:31.472918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:31.472919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:31.473048 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:31.473293 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:31.473294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:31.473294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:31.473603 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:31.473739 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:31.473924 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:55:31.474061 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260925 14:55:31.489836 15923 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:31.490500 15923 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:31.498958 15923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Generated new TSK 0
I20260925 14:55:31.499184 15923 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab.
Entry for principal oryx/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab.
I20260925 14:55:31.531697 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36855
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:31.663369 15939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:31.663592 15939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:31.663617 15939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:31.664909 15939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:31.664952 15939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:31.664983 15939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:31.666553 15939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:31.666829 15939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:31.667088 15939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:31.672871 15947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:31.672886 15945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:31.672863 15944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:31.673210 15939 server_base.cc:1034] running on GCE node
I20260925 14:55:31.673471 15939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:31.673678 15939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:31.674834 15939 hybrid_clock.cc:648] HybridClock initialized: now 1790348131674824 us; error 32 us; skew 500 ppm
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:31.677693 15939 init.cc:377] Logged in from keytab as oryx/127.9.141.65@KRBTEST.COM (short username oryx)
I20260925 14:55:31.678249 15939 webserver.cc:460] Webserver started at http://127.9.141.65:40797/ using document root <none> and password file <none>
I20260925 14:55:31.678519 15939 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:31.678596 15939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:31.678761 15939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:31.679962 15939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "1b722c1e25944aff96b7abe8ef020e95"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "f3cc6e3aa5dbad34807ef16c30fbed33"
server_key_iv: "fbc65df17d17128dab2e7c2e2cf5fadb"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.680445 15939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "1b722c1e25944aff96b7abe8ef020e95"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "f3cc6e3aa5dbad34807ef16c30fbed33"
server_key_iv: "fbc65df17d17128dab2e7c2e2cf5fadb"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.681960 15939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:31.682893 15954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.683109 15939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:31.683219 15939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "1b722c1e25944aff96b7abe8ef020e95"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "f3cc6e3aa5dbad34807ef16c30fbed33"
server_key_iv: "fbc65df17d17128dab2e7c2e2cf5fadb"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.683364 15939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:31.694938 15939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:31.697600 15939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:31.697789 15939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:31.698047 15939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:31.698397 15939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:31.698465 15939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.698511 15939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:31.698545 15939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.705062 15939 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:41787
I20260925 14:55:31.705117 16067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:41787 every 8 connection(s)
I20260925 14:55:31.705459 15939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:31.706035 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 15939
I20260925 14:55:31.706141 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:31.706439 9781 external_mini_cluster.cc:1428] Setting key d9e644108ff1871eaa54db461ad1c719
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.141.66@KRBTEST.COM; defaulting to no policy
I20260925 14:55:31.717197 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.706271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:33363 (local address 127.9.141.126:36855)
0925 14:55:31.706354 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:31.706356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:31.706564 (+ 208us) server_negotiation.cc:407] Connection header received
0925 14:55:31.707282 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:31.707283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:31.707296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:31.707378 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:31.708543 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.709049 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.710771 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.711024 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.711081 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:31.713903 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:31.713905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:31.713905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:31.713914 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:31.715544 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:31.716044 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:31.716045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:31.716046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:31.716111 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:31.716425 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:31.716426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:31.716426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:31.716586 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:31.716698 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:31.717001 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 14:55:31.717094 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:55:31.717418 16070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.706200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:33363)
0925 14:55:31.706425 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:31.706470 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:31.707155 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:31.707443 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:31.707447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:31.708075 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:31.708412 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:31.708421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.709143 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.709144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:31.710645 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:31.710648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.711122 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.711123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:31.711296 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:31.711915 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:31.711927 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:31.713749 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:31.715680 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:31.715688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:31.715691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:31.715941 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:31.716187 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:31.716188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:31.716189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:31.716329 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:31.716687 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:31.716695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:31.716850 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 14:55:31.716964 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:55:31.717150 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260925 14:55:31.718034 16068 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.66@KRBTEST.COM" created.
I20260925 14:55:31.718173 16068 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:31.718367 16068 heartbeater.cc:511] Master 127.9.141.126:36855 requested a full tablet report, sending...
I20260925 14:55:31.718946 15877 ts_manager.cc:194] Registered new tserver with Master: 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787)
I20260925 14:55:31.719853 15877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:33363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab.
Entry for principal oryx/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab.
I20260925 14:55:31.736509 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36855
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:31.818993 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:31.819216 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:31.819240 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:31.820426 16073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:31.820464 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:31.820492 16073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:31.822021 16073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:31.822220 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:31.822407 16073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:31.827533 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:31.827576 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:31.827538 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:31.827780 16073 server_base.cc:1034] running on GCE node
I20260925 14:55:31.827975 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:31.828192 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:31.829381 16073 hybrid_clock.cc:648] HybridClock initialized: now 1790348131829361 us; error 35 us; skew 500 ppm
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:31.837105 16073 init.cc:377] Logged in from keytab as oryx/127.9.141.66@KRBTEST.COM (short username oryx)
I20260925 14:55:31.837597 16073 webserver.cc:460] Webserver started at http://127.9.141.66:37755/ using document root <none> and password file <none>
I20260925 14:55:31.837865 16073 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:31.837937 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:31.838090 16073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:31.839255 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "83f0dee6f68d4e59958df94f0be61966"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "03e00e4d15a43badeae71c2c2b4e3f0e"
server_key_iv: "e9fc6e8ce497d7a9dca03c4991639b40"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.839771 16073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "83f0dee6f68d4e59958df94f0be61966"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "03e00e4d15a43badeae71c2c2b4e3f0e"
server_key_iv: "e9fc6e8ce497d7a9dca03c4991639b40"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.841670 16073 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 14:55:31.842670 16088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.842965 16073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:31.843111 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "83f0dee6f68d4e59958df94f0be61966"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "03e00e4d15a43badeae71c2c2b4e3f0e"
server_key_iv: "e9fc6e8ce497d7a9dca03c4991639b40"
server_key_version: "encryptionkey@0"
I20260925 14:55:31.843187 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:31.855782 16073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:31.858232 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:31.858413 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:31.858670 16073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:31.859014 16073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:31.859071 16073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.859110 16073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:31.859140 16073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:31.865545 16073 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:38681
I20260925 14:55:31.865584 16201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:38681 every 8 connection(s)
I20260925 14:55:31.865952 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:31.871809 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16073
I20260925 14:55:31.871912 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:31.872102 9781 external_mini_cluster.cc:1428] Setting key 29ca24673f8e1187c0cd360601641524
Sep 25 14:55:31 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:31.876921 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.866816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:41821 (local address 127.9.141.126:36855)
0925 14:55:31.866896 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:31.866899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:31.867189 (+ 290us) server_negotiation.cc:407] Connection header received
0925 14:55:31.867730 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:31.867731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:31.867741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:31.867799 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:31.869027 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.869594 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.870607 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.870812 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.870882 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:31.873878 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:31.873880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:31.873880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:31.873886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:31.875417 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:31.875910 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:31.875911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:31.875912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:31.875970 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:31.876229 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:31.876230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:31.876239 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:31.876353 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:31.876468 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:31.876745 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 14:55:31.876822 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:55:31.877126 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:31.866806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:41821)
0925 14:55:31.867038 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:31.867078 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:31.867635 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:31.867853 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:31.867857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:31.868704 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:31.868936 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:31.868941 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.869676 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.869677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:31.870511 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:31.870513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:31.870919 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:31.870920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:31.871029 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:31.871554 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:31.871562 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:31.873776 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:31.875541 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:31.875549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:31.875551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:31.875817 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:31.876045 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:31.876046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:31.876046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:31.876160 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:31.876461 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:31.876467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:31.876610 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:55:31.876722 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:55:31.876892 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
I20260925 14:55:31.877569 16202 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36855
I20260925 14:55:31.877663 16202 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:31.877868 16202 heartbeater.cc:511] Master 127.9.141.126:36855 requested a full tablet report, sending...
I20260925 14:55:31.878266 15877 ts_manager.cc:194] Registered new tserver with Master: 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66:38681)
I20260925 14:55:31.878700 15877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:41821
WARNING: no policy specified for oryx/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab.
Entry for principal oryx/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab.
I20260925 14:55:31.912019 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36855
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:31.994798 16207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:31.995018 16207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:31.995043 16207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:31.996287 16207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:31.996354 16207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:31.996403 16207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:31.997891 16207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:36855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:31.998093 16207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:31.998274 16207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:32.003310 16212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:32.003345 16213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:32.003492 16207 server_base.cc:1034] running on GCE node
W20260925 14:55:32.003527 16215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:32.003758 16207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:32.004019 16207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:32.005188 16207 hybrid_clock.cc:648] HybridClock initialized: now 1790348132005174 us; error 29 us; skew 500 ppm
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348132, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:32.007781 16207 init.cc:377] Logged in from keytab as oryx/127.9.141.67@KRBTEST.COM (short username oryx)
I20260925 14:55:32.008541 16207 webserver.cc:460] Webserver started at http://127.9.141.67:34599/ using document root <none> and password file <none>
I20260925 14:55:32.008796 16207 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:32.008867 16207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:32.009013 16207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:32.010174 16207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "94895020598b4f5d99ebab6732820bc7"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "bc6b3193ec4a037a6bdbcacba6bf31c9"
server_key_iv: "e5c0cb1cfdf20d9f99871c4519d58d5e"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.010540 16207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "94895020598b4f5d99ebab6732820bc7"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "bc6b3193ec4a037a6bdbcacba6bf31c9"
server_key_iv: "e5c0cb1cfdf20d9f99871c4519d58d5e"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.011852 16207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:32.012706 16222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.012883 16207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:32.012970 16207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "94895020598b4f5d99ebab6732820bc7"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "bc6b3193ec4a037a6bdbcacba6bf31c9"
server_key_iv: "e5c0cb1cfdf20d9f99871c4519d58d5e"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.013022 16207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:32.029675 16207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:32.031497 16207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:32.031605 16207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:32.031780 16207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:32.032094 16207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:32.032127 16207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.032151 16207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:32.032198 16207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.038261 16207 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:44533
I20260925 14:55:32.038304 16335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:44533 every 8 connection(s)
I20260925 14:55:32.038617 16207 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348132, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:32.047178 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16207
I20260925 14:55:32.047331 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:32.047588 9781 external_mini_cluster.cc:1428] Setting key 96411bb9c660295041f1e0e18c951be3
I20260925 14:55:32.048873 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.039323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:52771 (local address 127.9.141.126:36855)
0925 14:55:32.039379 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.039382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.039629 (+ 247us) server_negotiation.cc:407] Connection header received
0925 14:55:32.040098 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.040099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.040110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.040168 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.041323 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.041833 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.042851 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.043059 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.043125 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.045640 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.045642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.045642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.045648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.047158 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.047721 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.047722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.047723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.047789 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.048122 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.048123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.048123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.048255 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.048333 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.048692 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.048785 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:55:32.048995 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.039333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:52771)
0925 14:55:32.039507 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.039542 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.040006 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.040332 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.040336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.041015 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.041210 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.041214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.041934 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.041934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.042757 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.042759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.043163 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.043164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.043288 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.043850 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.043859 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.045542 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.047245 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.047252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.047255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.047614 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.047872 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.047873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.047874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.048053 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.048324 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.048330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.048536 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.048656 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.048799 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260925 14:55:32.049417 16336 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36855
I20260925 14:55:32.049500 16336 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:32.049681 16336 heartbeater.cc:511] Master 127.9.141.126:36855 requested a full tablet report, sending...
I20260925 14:55:32.050036 15877 ts_manager.cc:194] Registered new tserver with Master: 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533)
I20260925 14:55:32.050443 15877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:52771
I20260925 14:55:32.059616 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.141.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.125.keytab.
Entry for principal oryx/127.9.141.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.125.keytab.
I20260925 14:55:32.093196 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:34973
--webserver_interface=127.9.141.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.125.keytab
--principal=oryx/127.9.141.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36855,127.9.141.125:34973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:32.180331 16341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:32.180588 16341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:32.180610 16341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:32.181744 16341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:32.181779 16341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:32.181793 16341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:32.181806 16341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:32.181818 16341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:32.183245 16341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36855,127.9.141.125:34973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.125.keytab
--principal=oryx/127.9.141.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.141.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
W20260925 14:55:32.183346 16341 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.141.126:36855,127.9.141.125:34973'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:55:32.183460 16341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:32.183637 16341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:32.188803 16346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:32.188771 16349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:32.188902 16347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:32.189141 16341 server_base.cc:1034] running on GCE node
I20260925 14:55:32.189324 16341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:32.189618 16341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:32.190761 16341 hybrid_clock.cc:648] HybridClock initialized: now 1790348132190748 us; error 26 us; skew 500 ppm
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348132, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:32.193193 16341 init.cc:377] Logged in from keytab as oryx/127.9.141.125@KRBTEST.COM (short username oryx)
I20260925 14:55:32.193614 16341 webserver.cc:460] Webserver started at http://127.9.141.125:33795/ using document root <none> and password file <none>
I20260925 14:55:32.193837 16341 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:32.193882 16341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:32.194046 16341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:32.195186 16341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/instance:
uuid: "e78dcdba05fe4d0f90df66ae6e634261"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "f41dd36414ad26e63535864f19668fb8"
server_key_iv: "2a9a14f7b67c66e490b2dfb92b369553"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.195570 16341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal/instance:
uuid: "e78dcdba05fe4d0f90df66ae6e634261"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "f41dd36414ad26e63535864f19668fb8"
server_key_iv: "2a9a14f7b67c66e490b2dfb92b369553"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.196985 16341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:32.197789 16356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.198009 16341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:32.198110 16341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
uuid: "e78dcdba05fe4d0f90df66ae6e634261"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "f41dd36414ad26e63535864f19668fb8"
server_key_iv: "2a9a14f7b67c66e490b2dfb92b369553"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.198182 16341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:32.205991 16341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:32.208222 16341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:32.208390 16341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:32.212361 16341 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.125:34973
I20260925 14:55:32.212436 16408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.125:34973 every 8 connection(s)
I20260925 14:55:32.212893 16341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
I20260925 14:55:32.213894 16409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:32.215649 16409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:32.218663 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16341
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348132, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.125@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
I20260925 14:55:32.227442 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.216603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.9.141.126:36855)
0925 14:55:32.216669 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.216672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.216922 (+ 250us) server_negotiation.cc:407] Connection header received
0925 14:55:32.217697 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.217699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.217711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.217785 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.218401 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.219038 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.220682 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.220983 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.221059 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.223963 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.223964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.223965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.223972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.225724 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.226319 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.226320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.226321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.226386 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.226679 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.226680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.226680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.226860 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.226917 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.227246 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.227312 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:32.227687 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.216586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:33402)
0925 14:55:32.216777 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.216825 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.217569 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.217855 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.217859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.218120 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.218316 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.218319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.219147 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.219148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.220603 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.220605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.221089 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.221090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.221196 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.221710 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.221720 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.223862 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.225798 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.225803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.225804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.226218 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.226456 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.226457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.226458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.226602 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.226931 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.226936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.227103 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.227225 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.227380 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260925 14:55:32.228312 16409 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:32.228827 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.219197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.9.141.125:34973)
0925 14:55:32.219368 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.219371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.219401 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:55:32.219481 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.219483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.219504 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.219572 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.220270 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.221307 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.222308 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.222755 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.222840 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.225474 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.225482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.225482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.225488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.227292 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.227717 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.227720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.227721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.227804 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.228031 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.228032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.228032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.228232 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.228367 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.228630 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.228716 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348132, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.125@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
I20260925 14:55:32.235699 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.228625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:52872)
0925 14:55:32.228916 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.228925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.228967 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.229128 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.229129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.229418 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.229478 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.229480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.230091 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.230092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.230594 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.230596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.230959 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.230959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.231013 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.231430 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.231434 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.232620 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.234757 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.234758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.234758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.234919 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.235113 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.235114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.235114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.235166 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.235435 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.235435 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.235469 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.235515 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.235638 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260925 14:55:32.235827 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.228658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.9.141.125:34973)
0925 14:55:32.228894 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.228896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.229040 (+ 144us) server_negotiation.cc:407] Connection header received
0925 14:55:32.229056 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.229057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.229065 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.229128 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.229553 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.230021 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.230669 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.230879 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.230945 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.232703 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.232704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.232704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.232710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.234632 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.234992 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.234993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.234993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.235048 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.235243 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.235244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.235244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.235356 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.235438 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.235549 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.235688 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260925 14:55:32.237190 16409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Bootstrap starting.
I20260925 14:55:32.237823 16409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:32.238152 16409 log.cc:824] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:32.238935 16409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: No bootstrap required, opened a new log
I20260925 14:55:32.240679 16409 raft_consensus.cc:348] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } }
I20260925 14:55:32.240818 16409 raft_consensus.cc:374] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:32.240863 16409 raft_consensus.cc:729] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78dcdba05fe4d0f90df66ae6e634261, State: Initialized, Role: FOLLOWER
I20260925 14:55:32.240995 16409 consensus_queue.cc:260] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } }
I20260925 14:55:32.241310 16409 sys_catalog.cc:564] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:32.241314 16416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } } }
I20260925 14:55:32.241482 16416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:55:32.243149 16427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:32.243209 16427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:32.244910 16341 master_runner.cc:428] Detected that this master e78dcdba05fe4d0f90df66ae6e634261 is joining an existing cluster
I20260925 14:55:32.244979 16341 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.141.125:34973
I20260925 14:55:32.253435 16433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.246037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:33412)
0925 14:55:32.246180 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.246195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.246278 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.246469 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.246470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.246744 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.246857 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.246859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.247889 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.247890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.248615 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.248617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.249117 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.249118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.249184 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.249680 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.249686 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.250115 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.252204 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.252205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.252206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.252473 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.252695 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.252696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.252696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.252762 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.253067 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.253068 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.253114 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.253199 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.253317 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":42,"thread_start_us":81,"threads_started":1}
I20260925 14:55:32.253444 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.246073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.9.141.126:36855)
0925 14:55:32.246111 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.246113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.246251 (+ 138us) server_negotiation.cc:407] Connection header received
0925 14:55:32.246339 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.246341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.246361 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.246413 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.247034 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.247782 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.248701 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.248956 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.249033 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.250203 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.250205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.250206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.250213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.252111 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.252556 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.252558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.252558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.252618 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.252842 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.252843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.252844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.252991 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.253063 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.253253 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.253330 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:55:32.254612 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.246252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:52880)
0925 14:55:32.246396 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.246437 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.246494 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.246632 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.246633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.246885 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.246966 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.246968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.247888 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.247889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.248825 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.248828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.249842 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.249843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.249914 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.250487 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.250491 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.250856 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.253519 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.253519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.253520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.253766 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.253948 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.253949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.253949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.253984 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.254253 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.254254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.254298 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.254375 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.254476 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":29,"thread_start_us":58,"threads_started":1}
I20260925 14:55:32.255231 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.246387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.9.141.125:34973)
0925 14:55:32.246420 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.246422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.246503 (+ 81us) server_negotiation.cc:407] Connection header received
0925 14:55:32.246546 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.246546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.246555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.246608 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.247033 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.247805 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.249090 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.249755 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.249828 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.250947 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.250948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.250949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.250955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.253437 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.253842 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.253843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.253843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.253887 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.254049 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.254050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.254050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.254175 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.254241 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.254418 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.254540 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":19}
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.126@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
I20260925 14:55:32.264473 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.256158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.9.141.125:34973)
0925 14:55:32.256207 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.256209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.256585 (+ 376us) server_negotiation.cc:407] Connection header received
0925 14:55:32.256679 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.256679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.256688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.256733 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.257256 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.257945 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.259082 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.259268 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.259328 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.261661 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.261662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.261662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.261669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.263214 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.263612 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.263613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.263614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.263673 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.263913 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.263914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.263914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.264050 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.264118 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.264285 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.264360 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:55:32.264506 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.256108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:52894)
0925 14:55:32.256479 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.256525 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.256604 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.256740 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.256744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.257091 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.257151 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.257154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.258036 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.258037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.258988 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.258990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.259396 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.259398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.259483 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.259970 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.259976 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.261575 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.263310 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.263315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.263318 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.263535 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.263761 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.263762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.263763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.263826 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.264118 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.264123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.264173 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.264242 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.264346 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":76,"threads_started":1}
I20260925 14:55:32.265120 15877 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.141.125:34973: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261"
member_type: NON_VOTER
last_known_addr {
host: "127.9.141.125"
port: 34973
}
attrs {
promote: true
}
}
dest_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c"
cas_config_opid_index: -1
I20260925 14:55:32.265448 15877 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } }
I20260925 14:55:32.266111 16437 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e78dcdba05fe4d0f90df66ae6e634261 (127.9.141.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } } }
I20260925 14:55:32.266434 16383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:32.316402 16383 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Refusing update from remote peer dcbd9ff6bcbe4f428b3db34d35134b4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:55:32.316939 16437 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.141.125:34973
I20260925 14:55:32.317255 16341 master.cc:537] Master@127.9.141.125:34973 shutting down...
I20260925 14:55:32.317574 16438 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:32.317715 16436 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } } }
I20260925 14:55:32.317970 16436 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
W20260925 14:55:32.318717 15861 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 14:55:32.319217 15861 consensus_peers.cc:487] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c -> Peer e78dcdba05fe4d0f90df66ae6e634261 (127.9.141.125:34973): Couldn't send request to peer e78dcdba05fe4d0f90df66ae6e634261. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:55:32.322499 16341 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:32.322605 16341 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:32.322659 16341 tablet_replica.cc:330] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: stopping tablet replica
I20260925 14:55:32.325847 16341 master.cc:552] Master@127.9.141.125:34973 shutdown complete.
I20260925 14:55:32.325901 16341 master_runner.cc:305] Clearing existing system tablet
I20260925 14:55:32.326047 16341 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:32.326099 16341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:32.326848 16341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
WARNING: no policy specified for oryx/127.9.141.124@KRBTEST.COM; defaulting to no policy
I20260925 14:55:32.327392 16443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.327538 16341 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:32.327606 16341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.327632 16341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
uuid: "e78dcdba05fe4d0f90df66ae6e634261"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "f41dd36414ad26e63535864f19668fb8"
server_key_iv: "2a9a14f7b67c66e490b2dfb92b369553"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.328500 16341 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:32.452055 16341 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:55:32.452189 16341 log.cc:1196] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:55:32.463831 16341 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Deleting consensus metadata
I20260925 14:55:32.465605 16341 master_runner.cc:315] Copying system tablet from 127.9.141.126:36855
I20260925 14:55:32.466321 16341 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: tablet copy: Beginning tablet copy session from remote peer at address 127.9.141.126:36855
I20260925 14:55:32.472981 15916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.466804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.9.141.126:36855)
0925 14:55:32.466917 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.466919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.466930 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:32.467008 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.467009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.467044 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.467103 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:32.467491 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.468016 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.468917 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.469174 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.469275 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.470431 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:32.470432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:32.470432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:32.470439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:32.471921 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.472254 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.472255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.472255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.472300 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:32.472522 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:32.472522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:32.472522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:32.472633 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:32.472686 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.472859 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.472915 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 14:55:32.473028 16449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.466675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:33424)
0925 14:55:32.466854 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.466867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.466943 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.467143 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.467144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.467329 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:32.467376 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.467378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.468102 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.468102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.468806 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.468809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.469323 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.469324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.469393 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.470001 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:32.470007 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:32.470354 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:32.472008 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.472009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.472010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.472189 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.472356 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:32.472356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:32.472357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:32.472473 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:32.472715 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:32.472716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:32.472762 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.472839 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.472950 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260925 14:55:32.473420 15896 tablet_copy_service.cc:140] P dcbd9ff6bcbe4f428b3db34d35134b4c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e78dcdba05fe4d0f90df66ae6e634261 ({username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:33424)
I20260925 14:55:32.473507 15896 tablet_copy_service.cc:161] P dcbd9ff6bcbe4f428b3db34d35134b4c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e78dcdba05fe4d0f90df66ae6e634261 at {username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:33424: session id = e78dcdba05fe4d0f90df66ae6e634261-00000000000000000000000000000000
I20260925 14:55:32.474184 15896 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:55:32.474840 16341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:55:32.476068 16341 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: tablet copy: Starting download of 0 data blocks...
I20260925 14:55:32.476233 16341 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: tablet copy: Starting download of 1 WAL segments...
I20260925 14:55:32.477389 16341 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:55:32.553254 15896 tablet_copy_service.cc:342] P dcbd9ff6bcbe4f428b3db34d35134b4c: Request end of tablet copy session e78dcdba05fe4d0f90df66ae6e634261-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:33424
I20260925 14:55:32.553356 15896 tablet_copy_service.cc:433] P dcbd9ff6bcbe4f428b3db34d35134b4c: ending tablet copy session e78dcdba05fe4d0f90df66ae6e634261-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e78dcdba05fe4d0f90df66ae6e634261
I20260925 14:55:32.556356 16341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:32.557847 16450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:32.557930 16453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:32.558154 16451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:32.558183 16341 server_base.cc:1034] running on GCE node
I20260925 14:55:32.558377 16341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:32.558543 16341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:32.559667 16341 hybrid_clock.cc:648] HybridClock initialized: now 1790348132559660 us; error 26 us; skew 500 ppm
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.124@KRBTEST.COM" created.
Sep 25 14:55:32 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348132, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:32.563196 16341 init.cc:377] Logged in from keytab as oryx/127.9.141.125@KRBTEST.COM (short username oryx)
I20260925 14:55:32.563494 16341 webserver.cc:460] Webserver started at http://127.9.141.125:40653/ using document root <none> and password file <none>
I20260925 14:55:32.563644 16341 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:32.563684 16341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:32.564410 16341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:32.565059 16461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:32.565204 16341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:32.565264 16341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
uuid: "e78dcdba05fe4d0f90df66ae6e634261"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "f41dd36414ad26e63535864f19668fb8"
server_key_iv: "2a9a14f7b67c66e490b2dfb92b369553"
server_key_version: "encryptionkey@0"
I20260925 14:55:32.565330 16341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:32.585834 16341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:32.587628 16341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:32.590296 16341 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.125:34973
I20260925 14:55:32.591500 16513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.125:34973 every 8 connection(s)
I20260925 14:55:32.653353 16341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
I20260925 14:55:32.654824 16514 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:32.655596 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Bootstrap starting.
I20260925 14:55:32.658589 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:32.658758 16514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Bootstrap complete.
I20260925 14:55:32.658957 16514 raft_consensus.cc:348] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } }
I20260925 14:55:32.659062 16514 raft_consensus.cc:729] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e78dcdba05fe4d0f90df66ae6e634261, State: Initialized, Role: LEARNER
I20260925 14:55:32.659092 16514 consensus_queue.cc:260] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } }
I20260925 14:55:32.659250 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } } }
I20260925 14:55:32.659263 16514 sys_catalog.cc:564] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:32.659363 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:55:32.660579 16528 catalog_manager.cc:1259] Loaded cluster ID: 8daf7cc897dc4e01afca6554a3eb2ccf
I20260925 14:55:32.660689 16528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: loading cluster ID for follower catalog manager: success
I20260925 14:55:32.661616 16528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: acquiring CA information for follower catalog manager: success
I20260925 14:55:32.661899 16528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:55:32.721091 16068 heartbeater.cc:503] Master 127.9.141.126:36855 was elected leader, sending a full tablet report...
I20260925 14:55:32.807031 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.803861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:52904)
0925 14:55:32.804019 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:32.804040 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:32.804097 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:32.804367 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:32.804368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:32.804372 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:32.804453 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.804455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.805362 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.805363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.806193 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:32.806195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.806779 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.806781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:32.806834 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.806857 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:55:32.806910 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:55:32.806963 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260925 14:55:32.807056 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:32.804007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.9.141.125:34973)
0925 14:55:32.804153 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:32.804154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:32.804163 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:32.804188 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:32.804189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:32.804213 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:32.804305 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:32.804538 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.805252 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.806289 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:32.806688 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:32.806746 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:32.806800 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:32.806960 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 14:55:32.807000 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260925 14:55:32.807464 16488 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 14:55:32.807646 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbd9ff6bcbe4f428b3db34d35134b4c. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } } }
I20260925 14:55:32.807793 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:55:32.879549 16202 heartbeater.cc:503] Master 127.9.141.126:36855 was elected leader, sending a full tablet report...
I20260925 14:55:33.051316 16336 heartbeater.cc:503] Master 127.9.141.126:36855 was elected leader, sending a full tablet report...
I20260925 14:55:33.337486 16531 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: NON_VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: true } } }
I20260925 14:55:33.337671 16531 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:33.338064 16532 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: attempting to promote NON_VOTER e78dcdba05fe4d0f90df66ae6e634261 to VOTER
I20260925 14:55:33.338300 16532 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } }
I20260925 14:55:33.338809 16488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 LEARNER]: Refusing update from remote peer dcbd9ff6bcbe4f428b3db34d35134b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:55:33.339013 16532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:55:33.339980 16532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e78dcdba05fe4d0f90df66ae6e634261 (127.9.141.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } }
I20260925 14:55:33.340245 16488 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, e78dcdba05fe4d0f90df66ae6e634261 (127.9.141.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } }
I20260925 14:55:33.414211 16531 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } }
I20260925 14:55:33.414362 16531 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:33.414304 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } }
I20260925 14:55:33.414587 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:33.414726 16532 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } }
I20260925 14:55:33.414827 16532 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.124.keytab.
Entry for principal oryx/127.9.141.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.124.keytab.
W20260925 14:55:33.766928 9781 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.9.141.124: real 1.449s user 0.003s sys 0.000s
I20260925 14:55:33.767326 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:41017
--webserver_interface=127.9.141.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.124.keytab
--principal=oryx/127.9.141.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:33.859478 16537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:33.859695 16537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:33.859753 16537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:33.861353 16537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:33.861403 16537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:33.861420 16537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:33.861434 16537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:33.861447 16537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:33.863358 16537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.124.keytab
--principal=oryx/127.9.141.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:41017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.141.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:33.863634 16537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:33.863909 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:33.869300 16542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:33.869357 16543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:33.869532 16537 server_base.cc:1034] running on GCE node
W20260925 14:55:33.869560 16545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:33.869863 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:33.870155 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:33.871358 16537 hybrid_clock.cc:648] HybridClock initialized: now 1790348133871344 us; error 29 us; skew 500 ppm
Sep 25 14:55:33 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348133, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:33.874014 16537 init.cc:377] Logged in from keytab as oryx/127.9.141.124@KRBTEST.COM (short username oryx)
I20260925 14:55:33.874540 16537 webserver.cc:460] Webserver started at http://127.9.141.124:44447/ using document root <none> and password file <none>
I20260925 14:55:33.874792 16537 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:33.874859 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:33.875008 16537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:33.876123 16537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/instance:
uuid: "c314af0aa523494f8a3433efa62526b9"
format_stamp: "Formatted at 2026-09-25 14:55:33 on dist-test-slave-1sv7"
server_key: "d2ee1654e0f09c78dd42cea0eae4d53d"
server_key_iv: "0a3012f4762c474b472b9ac0b5eb6b61"
server_key_version: "encryptionkey@0"
I20260925 14:55:33.876591 16537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal/instance:
uuid: "c314af0aa523494f8a3433efa62526b9"
format_stamp: "Formatted at 2026-09-25 14:55:33 on dist-test-slave-1sv7"
server_key: "d2ee1654e0f09c78dd42cea0eae4d53d"
server_key_iv: "0a3012f4762c474b472b9ac0b5eb6b61"
server_key_version: "encryptionkey@0"
I20260925 14:55:33.877918 16537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:33.878779 16552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:33.878950 16537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:33.879045 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
uuid: "c314af0aa523494f8a3433efa62526b9"
format_stamp: "Formatted at 2026-09-25 14:55:33 on dist-test-slave-1sv7"
server_key: "d2ee1654e0f09c78dd42cea0eae4d53d"
server_key_iv: "0a3012f4762c474b472b9ac0b5eb6b61"
server_key_version: "encryptionkey@0"
I20260925 14:55:33.879127 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:33.896770 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:33.899080 16537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:33.899272 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:33.903887 16537 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.124:41017
I20260925 14:55:33.903919 16604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.124:41017 every 8 connection(s)
I20260925 14:55:33.904310 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
I20260925 14:55:33.905309 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:33.906843 16605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:55:33.912573 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16537
Sep 25 14:55:33 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348133, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.124@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:33.916759 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.907700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33436 (local address 127.9.141.126:36855)
0925 14:55:33.907839 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.907841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.907852 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:33.908364 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.908365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.908432 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.908517 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.909155 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.909698 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.910734 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.910941 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.911000 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.913559 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.913560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.913561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.913567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.915214 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.915640 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.915641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.915641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.915693 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.916024 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.916025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.916025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.916160 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.916233 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.916580 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.916665 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260925 14:55:33.916990 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.907504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:33436)
0925 14:55:33.907695 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.907763 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.908214 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.908589 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.908592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.908873 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.909065 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.909069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.909819 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.909820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.910644 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.910647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.911032 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.911033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.911143 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.911688 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.911695 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.913476 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.915284 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.915289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.915291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.915571 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.915761 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.915762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.915762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.915892 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.916260 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.916265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.916428 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.916555 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.916703 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260925 14:55:33.917505 16605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:55:33 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.141.124@KRBTEST.COM
Sep 25 14:55:33 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348133, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.124@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
I20260925 14:55:33.922372 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.913113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.9.141.124:41017)
0925 14:55:33.913258 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.913263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.913316 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:55:33.913388 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.913389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.913412 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.913475 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.914074 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.914730 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.915810 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.916338 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.916476 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.919240 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.919246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.919247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.919253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.920932 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.921317 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.921320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.921320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.921382 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.921563 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.921564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.921565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.921701 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.921847 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.922177 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.922253 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260925 14:55:33.924369 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.917748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:52914)
0925 14:55:33.917801 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.917810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.917848 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.918073 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.918073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.918204 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.918243 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.918245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.918939 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.918941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.919432 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.919434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.919834 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.919835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.919904 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.920446 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.920451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.921537 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.923375 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.923376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.923376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.923538 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.923757 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.923757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.923758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.923819 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.924067 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.924068 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.924108 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.924198 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.924288 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:55:33.924499 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.917823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.9.141.125:34973)
0925 14:55:33.917960 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.917962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.917968 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:55:33.917987 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.917988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.917996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.918070 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.918325 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.918836 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.919518 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.919742 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.919834 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.921633 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.921635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.921635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.921642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.923274 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.923624 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.923626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.923626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.923689 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.923891 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.923892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.923893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.923999 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.924074 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.924222 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.924349 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260925 14:55:33.924816 16605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:55:33 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348133, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.124@KRBTEST.COM for oryx/127.9.141.124@KRBTEST.COM
I20260925 14:55:33.930804 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.925049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.0.0.1:50604)
0925 14:55:33.925095 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.925105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.925139 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.925261 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.925261 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.925365 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.925390 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.925391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.925996 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.925996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.926459 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.926461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.926822 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.926823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.926858 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.927170 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.927174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.928334 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.929983 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.929984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.929984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.930136 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.930353 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.930353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.930354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.930404 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.930584 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.930584 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.930616 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.930672 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.930747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:55:33.930863 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.925118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.9.141.124:41017)
0925 14:55:33.925157 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.925159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.925166 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:33.925185 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.925185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.925194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.925256 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.925431 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.925927 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.926547 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.926747 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.926809 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.928477 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.928478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.928478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.928484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.929899 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.930249 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.930249 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.930250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.930294 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.930451 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.930452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.930452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.930531 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.930575 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.930688 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.930791 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":16}
I20260925 14:55:33.932220 16605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Bootstrap starting.
I20260925 14:55:33.933091 16605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:33.933384 16605 log.cc:824] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:33.934165 16605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: No bootstrap required, opened a new log
I20260925 14:55:33.935586 16605 raft_consensus.cc:348] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } }
I20260925 14:55:33.935705 16605 raft_consensus.cc:374] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:33.935726 16605 raft_consensus.cc:729] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c314af0aa523494f8a3433efa62526b9, State: Initialized, Role: FOLLOWER
I20260925 14:55:33.935875 16605 consensus_queue.cc:260] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } }
I20260925 14:55:33.936095 16613 sys_catalog.cc:455] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } } }
I20260925 14:55:33.936161 16605 sys_catalog.cc:564] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:33.936164 16613 sys_catalog.cc:458] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:55:33.937914 16624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:55:33.937968 16624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:55:33.940572 16537 master_runner.cc:428] Detected that this master c314af0aa523494f8a3433efa62526b9 is joining an existing cluster
I20260925 14:55:33.940613 16537 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.141.124:41017
I20260925 14:55:33.949334 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.941754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.9.141.125:34973)
0925 14:55:33.941808 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.941814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.942122 (+ 308us) server_negotiation.cc:407] Connection header received
0925 14:55:33.942217 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.942218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.942228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.942258 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.942789 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.943509 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944608 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.944923 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944960 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.946402 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.946404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.946404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.946412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.948129 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.948502 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.948503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.948504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.948564 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.948767 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.948769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.948769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.948907 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.948983 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.949150 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.949229 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:33.949330 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.941604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:52920)
0925 14:55:33.942062 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.942072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.942148 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.942336 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.942336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.942617 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.942689 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.942691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.943597 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.943598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.944515 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.944517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.945310 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.945310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.945354 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.945748 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.945753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.946323 (+ 570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.948211 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.948212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.948212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.948442 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.948633 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.948634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.948634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.948697 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.948984 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.948984 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.949032 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.949108 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.949214 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":50,"thread_start_us":72,"threads_started":1}
I20260925 14:55:33.974538 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15939
I20260925 14:55:33.976269 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.941994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.9.141.124:41017)
0925 14:55:33.942028 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.942029 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.942129 (+ 100us) server_negotiation.cc:407] Connection header received
0925 14:55:33.942214 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.942215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.942222 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.942250 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.942772 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.943508 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944603 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.944911 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944952 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.973539 (+ 28587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.973540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.973540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.973547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.975104 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.975526 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.975527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.975528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.975572 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.975738 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.975738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.975739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.975865 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.975906 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.976098 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.976174 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":18}
I20260925 14:55:33.976784 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.941932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.0.0.1:50610)
0925 14:55:33.942062 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.942072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.942148 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.942335 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.942336 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.942613 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.942690 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.942691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.943597 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.943598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.944515 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.944517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.945025 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.945026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.945080 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.973026 (+ 27946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.973031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.973439 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.975241 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.975242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.975242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.975452 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.975632 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.975633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.975633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.975678 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.975945 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.975946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.975991 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.976587 (+ 596us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.976690 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":33,"thread_start_us":62,"threads_started":1}
I20260925 14:55:33.978777 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.941586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:33446)
0925 14:55:33.941738 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.941750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.941815 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.942054 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.942054 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.942236 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.942333 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.942335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.943322 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.943322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.944189 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.944191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.944789 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.944858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.945301 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.945312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.972820 (+ 27508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.975396 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.975397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.975398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.975636 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.976275 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.976275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.976276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.978086 (+ 1810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.978456 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.978457 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.978499 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.978569 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.978671 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260925 14:55:33.979141 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.941774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.9.141.126:36855)
0925 14:55:33.941821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.941823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.941831 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:33.941875 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.941876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.941889 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.941947 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.942431 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.943210 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944287 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.944647 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.944698 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.972961 (+ 28263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.972964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.972965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.972975 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.974847 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.976138 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.976140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.976140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.976201 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.978183 (+ 1982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.978185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.978185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.978352 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.978427 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.978959 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.979051 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:33.980803 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:41787
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:36855
--webserver_port=40797
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:33 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348131, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.126@KRBTEST.COM for oryx/127.9.141.124@KRBTEST.COM
I20260925 14:55:33.990196 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.982191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.9.141.124:41017)
0925 14:55:33.982250 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:33.982252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:33.982334 (+ 82us) server_negotiation.cc:407] Connection header received
0925 14:55:33.982404 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:33.982405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:33.982415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:33.982479 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:33.982803 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.983550 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.984597 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.984885 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.984928 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.987252 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:33.987253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:33.987254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:33.987261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:33.988944 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.989368 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.989369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.989370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.989428 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:33.989651 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:33.989652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:33.989653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:33.989781 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:33.989848 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:33.990033 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:55:33.990106 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:33.990268 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:33.981550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.0.0.1:50626)
0925 14:55:33.982266 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:33.982278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:33.982342 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:33.982485 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:33.982486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:33.982684 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:33.982742 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.982744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.983643 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.983644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.984502 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:33.984505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:33.985004 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:33.985005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:33.985060 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:33.985499 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:33.985504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:33.987146 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:33.989032 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.989033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.989034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.989281 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.989506 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:33.989507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:33.989507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:33.989580 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:33.989857 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:33.989858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:33.989913 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 14:55:33.990000 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:55:33.990120 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":52,"threads_started":1}
I20260925 14:55:33.991390 15877 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.141.124:41017: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c314af0aa523494f8a3433efa62526b9"
member_type: NON_VOTER
last_known_addr {
host: "127.9.141.124"
port: 41017
}
attrs {
promote: true
}
}
dest_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c"
cas_config_opid_index: 6
I20260925 14:55:33.991552 15877 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } }
I20260925 14:55:33.992022 16488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Refusing update from remote peer dcbd9ff6bcbe4f428b3db34d35134b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:55:33.992167 16579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:33.992260 16633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:55:33.992944 16633 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c314af0aa523494f8a3433efa62526b9 (127.9.141.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:33.993146 16579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Refusing update from remote peer dcbd9ff6bcbe4f428b3db34d35134b4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:55:33.993908 16633 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.141.124:41017
I20260925 14:55:33.993981 16633 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:33.994135 16537 master.cc:537] Master@127.9.141.124:41017 shutting down...
I20260925 14:55:33.994361 16633 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:33.994424 16633 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:33.994530 16633 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:33.994571 16633 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:33.994841 16488 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c314af0aa523494f8a3433efa62526b9 (127.9.141.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:33.996080 16640 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:33.996222 16640 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:55:33.996886 15862 consensus_peers.cc:487] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c -> Peer c314af0aa523494f8a3433efa62526b9 (127.9.141.124:41017): Couldn't send request to peer c314af0aa523494f8a3433efa62526b9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:55:33.998071 16537 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:33.998163 16537 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:33.998210 16537 tablet_replica.cc:330] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: stopping tablet replica
I20260925 14:55:34.011075 16537 master.cc:552] Master@127.9.141.124:41017 shutdown complete.
I20260925 14:55:34.011201 16537 master_runner.cc:305] Clearing existing system tablet
I20260925 14:55:34.011401 16537 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:34.011503 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:34.012815 16537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:34.013415 16642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.013603 16537 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:34.013811 16537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:34.013890 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
uuid: "c314af0aa523494f8a3433efa62526b9"
format_stamp: "Formatted at 2026-09-25 14:55:33 on dist-test-slave-1sv7"
server_key: "d2ee1654e0f09c78dd42cea0eae4d53d"
server_key_iv: "0a3012f4762c474b472b9ac0b5eb6b61"
server_key_version: "encryptionkey@0"
I20260925 14:55:34.014763 16537 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:34.016505 16537 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:55:34.016637 16537 log.cc:1196] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:55:34.017083 16537 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Deleting consensus metadata
I20260925 14:55:34.017263 16537 master_runner.cc:315] Copying system tablet from 127.9.141.126:36855
I20260925 14:55:34.017920 16537 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: tablet copy: Beginning tablet copy session from remote peer at address 127.9.141.126:36855
I20260925 14:55:34.026005 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.018479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.9.141.126:36855)
0925 14:55:34.018525 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.018528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.018675 (+ 147us) server_negotiation.cc:407] Connection header received
0925 14:55:34.018784 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.018785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.018797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.018830 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.019260 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.020037 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.021276 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.021583 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.021625 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.022793 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.022794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.022795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.022801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.024705 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.025123 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.025124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.025125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.025183 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.025400 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.025400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.025401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.025549 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.025596 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.025812 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.025895 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:55:34.026805 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.018390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:33458)
0925 14:55:34.018595 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.018607 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.018704 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.018900 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.018901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.019101 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.019166 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.019168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.020137 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.020137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.021156 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.021158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.021703 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.021704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.021771 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.022279 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.022285 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.022688 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.024805 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.024806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.024807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.025029 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.025255 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.025256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.025256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.025315 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.025646 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.025647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.025695 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.026533 (+ 838us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.026671 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260925 14:55:34.029034 15896 tablet_copy_service.cc:140] P dcbd9ff6bcbe4f428b3db34d35134b4c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c314af0aa523494f8a3433efa62526b9 ({username='oryx', principal='oryx/127.9.141.124@KRBTEST.COM'} at 127.0.0.1:33458)
I20260925 14:55:34.029099 15896 tablet_copy_service.cc:161] P dcbd9ff6bcbe4f428b3db34d35134b4c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c314af0aa523494f8a3433efa62526b9 at {username='oryx', principal='oryx/127.9.141.124@KRBTEST.COM'} at 127.0.0.1:33458: session id = c314af0aa523494f8a3433efa62526b9-00000000000000000000000000000000
I20260925 14:55:34.029857 15896 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:55:34.030382 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:55:34.031744 16537 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: tablet copy: Starting download of 0 data blocks...
I20260925 14:55:34.031883 16537 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: tablet copy: Starting download of 1 WAL segments...
I20260925 14:55:34.033357 16537 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:55:34.034339 15896 tablet_copy_service.cc:342] P dcbd9ff6bcbe4f428b3db34d35134b4c: Request end of tablet copy session c314af0aa523494f8a3433efa62526b9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.141.124@KRBTEST.COM'} at 127.0.0.1:33458
I20260925 14:55:34.034404 15896 tablet_copy_service.cc:433] P dcbd9ff6bcbe4f428b3db34d35134b4c: ending tablet copy session c314af0aa523494f8a3433efa62526b9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c314af0aa523494f8a3433efa62526b9
I20260925 14:55:34.037710 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:34.039469 16649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.039746 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:34.039924 16537 server_base.cc:1034] running on GCE node
W20260925 14:55:34.048517 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:34.048748 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:34.048918 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:34.050024 16537 hybrid_clock.cc:648] HybridClock initialized: now 1790348134050013 us; error 26 us; skew 500 ppm
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:34.051617 16537 init.cc:377] Logged in from keytab as oryx/127.9.141.124@KRBTEST.COM (short username oryx)
I20260925 14:55:34.051901 16537 webserver.cc:460] Webserver started at http://127.9.141.124:43249/ using document root <none> and password file <none>
I20260925 14:55:34.052017 16537 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:34.052057 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:34.053022 16537 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:34.053583 16659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.053716 16537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:34.053759 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
uuid: "c314af0aa523494f8a3433efa62526b9"
format_stamp: "Formatted at 2026-09-25 14:55:33 on dist-test-slave-1sv7"
server_key: "d2ee1654e0f09c78dd42cea0eae4d53d"
server_key_iv: "0a3012f4762c474b472b9ac0b5eb6b61"
server_key_version: "encryptionkey@0"
I20260925 14:55:34.053843 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:34.077458 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:34.079726 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:34.082898 16537 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.124:41017
I20260925 14:55:34.082924 16711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.124:41017 every 8 connection(s)
I20260925 14:55:34.083225 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
I20260925 14:55:34.083985 16712 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:34.084444 16712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Bootstrap starting.
I20260925 14:55:34.086694 16712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:34.086925 16712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: Bootstrap complete.
I20260925 14:55:34.087114 16712 raft_consensus.cc:348] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } }
I20260925 14:55:34.087203 16712 raft_consensus.cc:729] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c314af0aa523494f8a3433efa62526b9, State: Initialized, Role: LEARNER
I20260925 14:55:34.087246 16712 consensus_queue.cc:260] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } }
I20260925 14:55:34.087427 16712 sys_catalog.cc:564] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:34.087421 16715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:34.087642 16715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:55:34.088950 16726 catalog_manager.cc:1259] Loaded cluster ID: 8daf7cc897dc4e01afca6554a3eb2ccf
I20260925 14:55:34.089040 16726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: loading cluster ID for follower catalog manager: success
I20260925 14:55:34.090325 16726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: acquiring CA information for follower catalog manager: success
I20260925 14:55:34.090620 16726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:55:34.099195 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:34.099462 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:34.099495 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:34.101243 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:34.101326 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:34.101365 16635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:34.103705 16635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:41787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=40797
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:34.103967 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:34.104257 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:34.109586 16731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.109663 16734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.109598 16732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:34.109916 16635 server_base.cc:1034] running on GCE node
I20260925 14:55:34.110196 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:34.110507 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:34.111675 16635 hybrid_clock.cc:648] HybridClock initialized: now 1790348134111657 us; error 31 us; skew 500 ppm
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:34.114207 16635 init.cc:377] Logged in from keytab as oryx/127.9.141.65@KRBTEST.COM (short username oryx)
I20260925 14:55:34.114640 16635 webserver.cc:460] Webserver started at http://127.9.141.65:40797/ using document root <none> and password file <none>
I20260925 14:55:34.114889 16635 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:34.114975 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:34.116510 16635 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:34.117120 16741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.117276 16635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.117372 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "1b722c1e25944aff96b7abe8ef020e95"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "f3cc6e3aa5dbad34807ef16c30fbed33"
server_key_iv: "fbc65df17d17128dab2e7c2e2cf5fadb"
server_key_version: "encryptionkey@0"
I20260925 14:55:34.117730 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:34.130007 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:34.131963 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:34.132119 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:34.132421 16635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:34.132720 16635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:34.132776 16635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.132817 16635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:34.132843 16635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.138571 16635 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:41787
I20260925 14:55:34.138664 16854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:41787 every 8 connection(s)
I20260925 14:55:34.139006 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:34.141630 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16635
I20260925 14:55:34.141775 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16073
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.124@KRBTEST.COM
I20260925 14:55:34.148257 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:38681
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:36855
--webserver_port=37755
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:34.152242 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.140013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:35979 (local address 127.9.141.124:41017)
0925 14:55:34.140214 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.140216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.140226 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:34.141260 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.141262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.141288 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.141353 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.143074 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.144222 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145463 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.145830 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145872 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.148313 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.148314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.148315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.148322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.150549 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151042 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.151043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.151044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.151102 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151440 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.151441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.151441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.151582 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.151652 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.151996 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.152094 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260925 14:55:34.152339 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.139743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.9.141.65:37515)
0925 14:55:34.140026 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.140037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.141179 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.141472 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.141473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.141926 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.142172 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.142178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.143395 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.143396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.144761 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.144764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.145277 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.145421 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.146044 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.146054 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.148750 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.150946 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.150947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.150947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.151158 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.151343 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.151344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.151430 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.151710 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.151711 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.151831 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.151933 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.152103 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":95,"thread_start_us":89,"threads_started":1}
I20260925 14:55:34.152632 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.139833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:52949 (local address 127.9.141.126:36855)
0925 14:55:34.140119 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.140121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.140318 (+ 197us) server_negotiation.cc:407] Connection header received
0925 14:55:34.141306 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.141307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.141319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.141411 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.142639 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.143375 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.144875 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.145164 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145213 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.149508 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.149509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.149509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.149517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.151389 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151720 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.151721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.151721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.151776 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151984 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.151985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.151986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.152112 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.152170 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.152461 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.152542 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260925 14:55:34.152787 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.140000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:37515 (local address 127.9.141.125:34973)
0925 14:55:34.140046 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.140048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.140112 (+ 64us) server_negotiation.cc:407] Connection header received
0925 14:55:34.141348 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.141350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.141360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.141407 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.142465 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.143279 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.144863 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.145162 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145218 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.148841 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.148842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.148843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.148850 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.150849 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151218 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.151219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.151219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.151275 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151505 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.151506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.151506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.151628 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.151682 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.152610 (+ 928us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.152690 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:34.152866 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.139743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:52949)
0925 14:55:34.140242 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.140252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.141179 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.141478 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.141479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.142487 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.142549 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.142550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.143462 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.143463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.144761 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.144764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.145290 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.145421 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.146148 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.146152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.149404 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.151468 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.151469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.151652 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.151844 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.151845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.151917 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.152195 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.152196 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.152231 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.152280 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.152366 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":51,"thread_start_us":86,"threads_started":1}
I20260925 14:55:34.153131 16857 heartbeater.cc:347] Connected to a master server at 127.9.141.125:34973
I20260925 14:55:34.153234 16857 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.153304 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.139743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.9.141.65:35979)
0925 14:55:34.139972 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.140013 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.141158 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.141436 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.141440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.142895 (+ 1455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.142972 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.142974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.144323 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.144324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.145364 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.145367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.145961 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.145962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.146050 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.146453 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.146458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.148204 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.150649 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.150655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.150657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.150973 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.151181 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.151182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.151182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.151319 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.151654 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.151661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.151831 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.152371 (+ 540us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.152519 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"spinlock_wait_cycles":8448,"thread_start_us":134,"threads_started":1}
I20260925 14:55:34.153450 16857 heartbeater.cc:511] Master 127.9.141.125:34973 requested a full tablet report, sending...
I20260925 14:55:34.153916 16478 ts_manager.cc:194] Registered new tserver with Master: 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787)
I20260925 14:55:34.154045 16855 heartbeater.cc:347] Connected to a master server at 127.9.141.124:41017
I20260925 14:55:34.154059 16856 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36855
I20260925 14:55:34.154151 16856 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.154227 16856 heartbeater.cc:511] Master 127.9.141.126:36855 requested a full tablet report, sending...
I20260925 14:55:34.154106 16855 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.154326 16855 heartbeater.cc:511] Master 127.9.141.124:41017 requested a full tablet report, sending...
I20260925 14:55:34.154498 15877 ts_manager.cc:194] Re-registered known tserver with Master: 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787)
I20260925 14:55:34.154657 16676 ts_manager.cc:194] Registered new tserver with Master: 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787)
I20260925 14:55:34.154971 15877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:52949
W20260925 14:55:34.234082 16863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:34.234310 16863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:34.234349 16863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:34.235503 16863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:34.235571 16863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:34.235617 16863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:34.237210 16863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=37755
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:34.237483 16863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:34.237712 16863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:34.242794 16868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.242815 16869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.242807 16871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:34.243072 16863 server_base.cc:1034] running on GCE node
I20260925 14:55:34.243273 16863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:34.243464 16863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:34.244656 16863 hybrid_clock.cc:648] HybridClock initialized: now 1790348134244646 us; error 40 us; skew 500 ppm
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:34.247279 16863 init.cc:377] Logged in from keytab as oryx/127.9.141.66@KRBTEST.COM (short username oryx)
I20260925 14:55:34.247673 16863 webserver.cc:460] Webserver started at http://127.9.141.66:37755/ using document root <none> and password file <none>
I20260925 14:55:34.247860 16863 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:34.247910 16863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:34.249248 16863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:34.249778 16878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.249964 16863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:34.250030 16863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "83f0dee6f68d4e59958df94f0be61966"
format_stamp: "Formatted at 2026-09-25 14:55:31 on dist-test-slave-1sv7"
server_key: "03e00e4d15a43badeae71c2c2b4e3f0e"
server_key_iv: "e9fc6e8ce497d7a9dca03c4991639b40"
server_key_version: "encryptionkey@0"
I20260925 14:55:34.250329 16863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:34.260836 16863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:34.262719 16863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:34.262825 16863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:34.262988 16863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:34.263290 16863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:34.263324 16863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.263346 16863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:34.263392 16863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.269423 16863 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:38681
I20260925 14:55:34.269495 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:38681 every 8 connection(s)
I20260925 14:55:34.269816 16863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:34.275609 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16863
I20260925 14:55:34.275772 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16207
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.124@KRBTEST.COM
I20260925 14:55:34.280450 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.270743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:53387 (local address 127.9.141.125:34973)
0925 14:55:34.270786 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.270788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.270908 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:55:34.271390 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.271392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.271403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.271438 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.272467 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.273156 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274105 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.274299 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274384 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.277517 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.277519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.277519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.277526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.279037 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279472 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.279474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.279474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.279531 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279787 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.279788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.279788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.279910 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.279986 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.280257 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.280330 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:34.280666 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.270769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:50631 (local address 127.9.141.124:41017)
0925 14:55:34.270812 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.270814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.270824 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:34.271373 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.271374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.271388 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.271450 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.272093 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.272793 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.273894 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.274158 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274217 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.277791 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.277792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.277792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.277799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.279381 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279776 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.279777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.279778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.279830 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.280014 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.280014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.280015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.280146 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.280187 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.280421 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.280516 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:55:34.280915 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.270490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.9.141.66:53387)
0925 14:55:34.270708 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.270841 (+ 133us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.271294 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.271580 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.271581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.271818 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.271971 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.271978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.273239 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.273239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.274012 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.274013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.274385 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.274448 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.275165 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.275170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.277395 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.279139 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.279148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.279377 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.279596 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.279597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.279711 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.279978 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.279984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.280129 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.280240 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.280418 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260925 14:55:34.281082 16997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.270490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.9.141.66:50631)
0925 14:55:34.270702 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.270744 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.271275 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.271521 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.271524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.271783 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.271970 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.271975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.272899 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.272900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.273791 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.273793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.274277 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.274427 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.275111 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.275115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.277694 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.279471 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.279472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.279700 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.279896 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.279896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.279897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.279952 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.280231 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.280232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.280274 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.280785 (+ 511us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.280896 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":49,"thread_start_us":75,"threads_started":1}
I20260925 14:55:34.281409 16994 heartbeater.cc:347] Connected to a master server at 127.9.141.125:34973
I20260925 14:55:34.281507 16994 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.281680 16994 heartbeater.cc:511] Master 127.9.141.125:34973 requested a full tablet report, sending...
I20260925 14:55:34.281415 16992 heartbeater.cc:347] Connected to a master server at 127.9.141.124:41017
I20260925 14:55:34.281870 16992 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.281931 16992 heartbeater.cc:511] Master 127.9.141.124:41017 requested a full tablet report, sending...
I20260925 14:55:34.282106 16478 ts_manager.cc:194] Registered new tserver with Master: 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66:38681)
I20260925 14:55:34.282213 16676 ts_manager.cc:194] Registered new tserver with Master: 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66:38681)
I20260925 14:55:34.282619 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:44533
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:36855
--webserver_port=34599
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:34.285943 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.270490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:34001)
0925 14:55:34.270960 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.270968 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.271294 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.271911 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.271912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.272096 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.272153 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.272154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.272897 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.272898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.273954 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.273956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.274451 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.274505 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.275088 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.275097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.283293 (+ 8196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.285041 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.285041 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.285042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.285196 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.285396 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.285397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.285397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.285443 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.285707 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.285708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.285742 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.285797 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.285870 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":44,"spinlock_wait_cycles":26368,"thread_start_us":103,"threads_started":1}
I20260925 14:55:34.286069 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.270590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:34001 (local address 127.9.141.126:36855)
0925 14:55:34.270766 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.270768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.271048 (+ 280us) server_negotiation.cc:407] Connection header received
0925 14:55:34.271400 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.271401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.271413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.271444 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.272236 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.272792 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274062 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.274341 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.274401 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.283377 (+ 8976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.283379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.283379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.283388 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.284961 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.285269 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.285271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.285271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.285331 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.285503 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.285504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.285504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.285642 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.285709 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.285843 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.285927 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260925 14:55:34.286422 16993 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36855
I20260925 14:55:34.286547 16993 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.286638 16993 heartbeater.cc:511] Master 127.9.141.126:36855 requested a full tablet report, sending...
I20260925 14:55:34.286944 15877 ts_manager.cc:194] Re-registered known tserver with Master: 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66:38681)
I20260925 14:55:34.287436 15877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:34001
W20260925 14:55:34.365082 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:34.365278 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:34.365299 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:34.366672 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:34.366714 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:34.366740 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:34.368323 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:44533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=34599
--tserver_master_addrs=127.9.141.126:36855,127.9.141.125:34973,127.9.141.124:41017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:34.368621 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:34.368866 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:34.373828 17005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.373848 17004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:34.373907 17007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:34.374066 16999 server_base.cc:1034] running on GCE node
I20260925 14:55:34.374308 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:34.374554 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:34.375725 16999 hybrid_clock.cc:648] HybridClock initialized: now 1790348134375712 us; error 30 us; skew 500 ppm
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:34.378290 16999 init.cc:377] Logged in from keytab as oryx/127.9.141.67@KRBTEST.COM (short username oryx)
I20260925 14:55:34.378844 16999 webserver.cc:460] Webserver started at http://127.9.141.67:34599/ using document root <none> and password file <none>
I20260925 14:55:34.379118 16999 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:34.379195 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:34.380977 16999 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:34.381753 17014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.381979 16999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:34.382122 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "94895020598b4f5d99ebab6732820bc7"
format_stamp: "Formatted at 2026-09-25 14:55:32 on dist-test-slave-1sv7"
server_key: "bc6b3193ec4a037a6bdbcacba6bf31c9"
server_key_iv: "e5c0cb1cfdf20d9f99871c4519d58d5e"
server_key_version: "encryptionkey@0"
I20260925 14:55:34.382467 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:34.391702 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:34.393826 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:34.393988 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:34.394232 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:34.394531 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:34.394585 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.394625 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:34.394675 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:34.400741 16999 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:44533
I20260925 14:55:34.400790 17127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:44533 every 8 connection(s)
I20260925 14:55:34.401149 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.125@KRBTEST.COM
I20260925 14:55:34.407931 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 16999
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.124@KRBTEST.COM
Sep 25 14:55:34 dist-test-slave-1sv7 krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348134, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:34.412214 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.401827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.9.141.67:36713)
0925 14:55:34.402154 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.402200 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.402787 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.403109 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.403110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.403451 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.403638 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.403643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.404643 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.404644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.405561 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.405563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.406021 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.406022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.406088 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.406640 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.406645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.408118 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.410443 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.410450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.410453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.410759 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.411012 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.411013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.411156 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.411455 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.411462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.411627 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.411736 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.411911 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":9,"thread_start_us":92,"threads_started":1}
I20260925 14:55:34.412290 16862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.402133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:51907 (local address 127.9.141.124:41017)
0925 14:55:34.402221 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.402223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.402283 (+ 60us) server_negotiation.cc:407] Connection header received
0925 14:55:34.402895 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.402896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.402908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.402969 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.403752 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.404563 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.405472 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.405745 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.405813 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.408779 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.408780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.408780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.408788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.410959 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411357 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.411358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.411358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.411416 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411622 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.411623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.411624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.411734 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.411789 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.412108 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.412174 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:55:34.412563 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.402132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:36713 (local address 127.9.141.125:34973)
0925 14:55:34.402183 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.402186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.402295 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:55:34.403023 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.403024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.403035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.403088 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.403751 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.404564 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.405666 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.405942 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.406022 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.408220 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.408222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.408223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.408232 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.410343 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.410858 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.410859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.410859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.410921 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411230 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.411231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.411232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.411367 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.411429 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.412361 (+ 932us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.412460 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:55:34.412837 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.401827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:55185)
0925 14:55:34.402381 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.402391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.402799 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.403065 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.403071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.403434 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.403637 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.403642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.404355 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.404356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.405566 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.405568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.406140 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.406141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.406206 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.406810 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.406814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.409319 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.411665 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.411666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.411855 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.412059 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.412059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.412060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.412130 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.412553 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.412554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.412593 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.412658 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.412743 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":52,"thread_start_us":78,"threads_started":1}
I20260925 14:55:34.412885 17130 heartbeater.cc:347] Connected to a master server at 127.9.141.125:34973
I20260925 14:55:34.412235 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.401827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.9.141.67:51907)
0925 14:55:34.402202 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.402211 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.402787 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.403442 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.403443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.403597 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:34.403641 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.403643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.404662 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.404663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.405373 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.405375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.405829 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.405830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.405965 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.406462 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:34.406470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:34.408672 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:34.411052 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.411053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.411270 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.411494 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:34.411495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:34.411548 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:34.411812 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:34.411812 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:34.411854 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.411912 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.412005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260925 14:55:34.412987 17130 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.413169 17130 heartbeater.cc:511] Master 127.9.141.125:34973 requested a full tablet report, sending...
I20260925 14:55:34.413339 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.401877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:55185 (local address 127.9.141.126:36855)
0925 14:55:34.401931 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.401933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.402452 (+ 519us) server_negotiation.cc:407] Connection header received
0925 14:55:34.402924 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.402925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.402937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.402969 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.403737 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.404247 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.405717 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.406003 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.406073 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.409408 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.409410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.409411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.409420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.411585 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.411932 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.411933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.411934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.411990 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.412281 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.412282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.412282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.412462 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.412527 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.413161 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.413245 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:55:34.413594 16478 ts_manager.cc:194] Registered new tserver with Master: 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533)
I20260925 14:55:34.413751 17128 heartbeater.cc:347] Connected to a master server at 127.9.141.124:41017
I20260925 14:55:34.413810 17128 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.413869 17128 heartbeater.cc:511] Master 127.9.141.124:41017 requested a full tablet report, sending...
I20260925 14:55:34.414064 17129 heartbeater.cc:347] Connected to a master server at 127.9.141.126:36855
I20260925 14:55:34.414104 17129 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:34.414157 17129 heartbeater.cc:511] Master 127.9.141.126:36855 requested a full tablet report, sending...
I20260925 14:55:34.414147 16676 ts_manager.cc:194] Registered new tserver with Master: 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533)
I20260925 14:55:34.414359 15877 ts_manager.cc:194] Re-registered known tserver with Master: 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533)
I20260925 14:55:34.414839 15877 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:55185
I20260925 14:55:34.419369 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.409749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.9.141.124:41017)
0925 14:55:34.410232 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.410234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.410504 (+ 270us) server_negotiation.cc:407] Connection header received
0925 14:55:34.410526 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.410526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.410538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.410591 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.411411 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.412174 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.414181 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.414524 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.414588 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.416459 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.416460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.416460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.416467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.418072 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.418528 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.418529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.418529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.418590 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.418795 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.418795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.418796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.418925 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.419013 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.419195 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.419274 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":54,"threads_started":1}
I20260925 14:55:34.421810 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.409513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.9.141.125:34973)
0925 14:55:34.409993 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.409995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.412588 (+ 2593us) server_negotiation.cc:407] Connection header received
0925 14:55:34.413856 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.413857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.413868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.413917 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.414498 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.415281 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.416204 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.416473 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.416564 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.418508 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.418509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.418509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.418515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.420639 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.421060 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.421061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.421062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.421111 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.421305 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.421306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.421306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.421389 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.421467 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.421622 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.421698 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":62,"threads_started":1}
I20260925 14:55:34.422802 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.410207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.9.141.126:36855)
0925 14:55:34.413408 (+ 3201us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.413410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.413417 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:34.413437 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.413438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.413461 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.413521 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.414930 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.415946 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.417736 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.418075 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.418149 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.419911 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.419912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.419912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.419918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.421666 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.422031 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.422032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.422033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.422085 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.422291 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.422292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.422292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.422444 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.422545 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.422688 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.422745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3187,"thread_start_us":91,"threads_started":1}
I20260925 14:55:34.424329 15877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:34.425907 15877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:34.433717 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.429879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.9.141.65:41787)
0925 14:55:34.430216 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.430220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.430228 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:34.430261 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.430263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.430281 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.430354 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:34.430509 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.431328 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.432693 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.433358 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.433447 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.433520 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.433606 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.433641 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":77,"threads_started":1}
I20260925 14:55:34.433706 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.429721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:41787 (local address 127.0.0.1:54460)
0925 14:55:34.429897 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.429906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.429948 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.430361 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.430362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.430363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:34.430419 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.430420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.431432 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.431433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.432575 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.432585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.433446 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.433447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.433522 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.433527 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.433586 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.433629 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260925 14:55:34.433913 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.429721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:44533 (local address 127.0.0.1:37418)
0925 14:55:34.429772 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.429784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.429821 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.430352 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.430354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.430355 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:34.430419 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.430420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.431692 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.431693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.432775 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.432776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.433628 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.433629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.433703 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.433707 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.433736 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.433820 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":4}
I20260925 14:55:34.434401 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.430037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.9.141.67:44533)
0925 14:55:34.430174 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.430179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.430190 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:34.430229 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.430232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.430249 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.430343 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:34.430495 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.431606 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.432834 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.433512 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.434016 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.434096 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.434127 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.434167 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260925 14:55:34.434370 16789 tablet_service.cc:1467] Processing CreateTablet for tablet b36ba764a0534e9aab7be137090a10d1 (DEFAULT_TABLE table=test-table [id=b6069bbf8c6b4e6b91dc0c2da0abd489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:34.434635 16789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36ba764a0534e9aab7be137090a10d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:34.434767 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.429773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:38681 (local address 127.0.0.1:33316)
0925 14:55:34.429902 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.429911 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.430048 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.430633 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.430634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.430635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:34.430667 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.430669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.431826 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.431827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.432671 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.432674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.434516 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.434517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.434595 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.434601 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.434657 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.434696 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 14:55:34.435079 17062 tablet_service.cc:1467] Processing CreateTablet for tablet b36ba764a0534e9aab7be137090a10d1 (DEFAULT_TABLE table=test-table [id=b6069bbf8c6b4e6b91dc0c2da0abd489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:34.435349 17062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36ba764a0534e9aab7be137090a10d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:34.436868 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.429981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.9.141.66:38681)
0925 14:55:34.430490 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.430495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.430504 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:34.430541 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.430543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.430558 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.430640 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:34.430726 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.431737 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.432764 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.433531 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.433591 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.433662 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.436741 (+ 3079us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.436792 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":68,"threads_started":1}
I20260925 14:55:34.437459 17154 tablet_bootstrap.cc:492] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Bootstrap starting.
I20260925 14:55:34.438109 17154 tablet_bootstrap.cc:654] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:34.438184 17155 tablet_bootstrap.cc:492] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Bootstrap starting.
I20260925 14:55:34.438475 17154 log.cc:824] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:34.438647 16926 tablet_service.cc:1467] Processing CreateTablet for tablet b36ba764a0534e9aab7be137090a10d1 (DEFAULT_TABLE table=test-table [id=b6069bbf8c6b4e6b91dc0c2da0abd489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:34.438772 17155 tablet_bootstrap.cc:654] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:34.438973 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36ba764a0534e9aab7be137090a10d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:34.439055 17155 log.cc:824] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:34.439509 17154 tablet_bootstrap.cc:492] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: No bootstrap required, opened a new log
I20260925 14:55:34.439580 17154 ts_tablet_manager.cc:1397] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:34.439845 17155 tablet_bootstrap.cc:492] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: No bootstrap required, opened a new log
I20260925 14:55:34.439913 17155 ts_tablet_manager.cc:1397] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:34.441234 17154 raft_consensus.cc:348] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.441392 17154 raft_consensus.cc:374] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:34.441450 17154 raft_consensus.cc:729] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b722c1e25944aff96b7abe8ef020e95, State: Initialized, Role: FOLLOWER
I20260925 14:55:34.441561 17154 consensus_queue.cc:260] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.441854 17154 ts_tablet_manager.cc:1428] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:34.441946 16856 heartbeater.cc:503] Master 127.9.141.126:36855 was elected leader, sending a full tablet report...
I20260925 14:55:34.441994 17155 raft_consensus.cc:348] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.442145 17155 raft_consensus.cc:374] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:34.442185 17155 raft_consensus.cc:729] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94895020598b4f5d99ebab6732820bc7, State: Initialized, Role: FOLLOWER
I20260925 14:55:34.442313 17155 consensus_queue.cc:260] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.442550 17155 ts_tablet_manager.cc:1428] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:34.442893 17129 heartbeater.cc:503] Master 127.9.141.126:36855 was elected leader, sending a full tablet report...
I20260925 14:55:34.444072 17160 tablet_bootstrap.cc:492] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Bootstrap starting.
I20260925 14:55:34.444700 17160 tablet_bootstrap.cc:654] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:34.444980 17160 log.cc:824] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:34.445605 17160 tablet_bootstrap.cc:492] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: No bootstrap required, opened a new log
I20260925 14:55:34.445685 17160 ts_tablet_manager.cc:1397] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:55:34.447031 17160 raft_consensus.cc:348] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.447175 17160 raft_consensus.cc:374] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:34.447219 17160 raft_consensus.cc:729] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f0dee6f68d4e59958df94f0be61966, State: Initialized, Role: FOLLOWER
I20260925 14:55:34.447345 17160 consensus_queue.cc:260] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.447579 17160 ts_tablet_manager.cc:1428] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:55:34.447654 16993 heartbeater.cc:503] Master 127.9.141.126:36855 was elected leader, sending a full tablet report...
W20260925 14:55:34.520789 16995 tablet.cc:2369] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:34.611581 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.608931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.9.141.124:41017)
0925 14:55:34.609003 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.609005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.609034 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:55:34.609072 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.609073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.609104 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.609180 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:34.609346 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.609895 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.610821 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.611247 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.611346 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.611439 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.611458 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.611497 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:55:34.611581 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.608842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.0.0.1:50644)
0925 14:55:34.608907 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.608919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.608995 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.609183 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.609184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.609185 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:34.609240 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.609242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.609989 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.609991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.610721 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.610724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.611340 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.611341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.611398 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.611402 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.611444 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.611495 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:55:34.612125 16686 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 14:55:34.613207 17163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcbd9ff6bcbe4f428b3db34d35134b4c. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:34.613363 17163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: This master's current role is: LEARNER
W20260925 14:55:34.640789 16858 tablet.cc:2369] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:34.652005 17131 tablet.cc:2369] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:34.716478 17162 raft_consensus.cc:482] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:34.716602 17162 raft_consensus.cc:504] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.717103 17162 leader_election.cc:288] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533), 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787)
I20260925 14:55:34.720166 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.717386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:37945 (local address 127.9.141.67:44533)
0925 14:55:34.717439 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.717441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.717450 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:34.717480 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.717481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.717489 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.717559 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:34.717738 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.718253 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.719377 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.719775 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.719873 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.719880 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.720046 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.720088 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:55:34.720177 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.717289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:44533 (local address 127.9.141.66:37945)
0925 14:55:34.717345 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.717359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.717421 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.717566 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.717566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.717568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:34.717651 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.717654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.718334 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.718335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.719290 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.719292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.719877 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.719878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.719969 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.719973 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.720040 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.720075 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":24}
I20260925 14:55:34.720271 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.717380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:45805 (local address 127.9.141.65:41787)
0925 14:55:34.717426 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.717428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.717463 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:55:34.717563 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.717564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.717574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.717631 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:34.717815 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.718346 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.719418 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.719900 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.719976 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.719985 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.720158 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.720197 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:55:34.720501 16996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.717296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:41787 (local address 127.9.141.66:45805)
0925 14:55:34.717410 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:34.717421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:34.717476 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:34.717643 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:34.717643 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:34.717645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:34.717733 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.717735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.718430 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.718430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.719281 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:34.719284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.719981 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.719982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:34.720070 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.720073 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:34.720131 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:55:34.720167 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":71}
I20260925 14:55:34.720922 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36ba764a0534e9aab7be137090a10d1" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b722c1e25944aff96b7abe8ef020e95" is_pre_election: true
I20260925 14:55:34.720934 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36ba764a0534e9aab7be137090a10d1" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94895020598b4f5d99ebab6732820bc7" is_pre_election: true
I20260925 14:55:34.721096 16809 raft_consensus.cc:2454] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 0.
I20260925 14:55:34.721103 17082 raft_consensus.cc:2454] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 0.
I20260925 14:55:34.721323 16881 leader_election.cc:302] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f0dee6f68d4e59958df94f0be61966, 94895020598b4f5d99ebab6732820bc7; no voters:
I20260925 14:55:34.721490 17162 raft_consensus.cc:2781] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:34.721565 17162 raft_consensus.cc:482] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:34.721601 17162 raft_consensus.cc:3037] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:34.722388 17162 raft_consensus.cc:504] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.722556 17162 leader_election.cc:288] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 election: Requested vote from peers 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533), 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787)
I20260925 14:55:34.722669 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36ba764a0534e9aab7be137090a10d1" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94895020598b4f5d99ebab6732820bc7"
I20260925 14:55:34.722713 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36ba764a0534e9aab7be137090a10d1" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b722c1e25944aff96b7abe8ef020e95"
I20260925 14:55:34.722761 17082 raft_consensus.cc:3037] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:34.722782 16809 raft_consensus.cc:3037] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:34.723472 16809 raft_consensus.cc:2454] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 1.
I20260925 14:55:34.723475 17082 raft_consensus.cc:2454] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 1.
I20260925 14:55:34.723655 16879 leader_election.cc:302] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b722c1e25944aff96b7abe8ef020e95, 83f0dee6f68d4e59958df94f0be61966; no voters:
I20260925 14:55:34.723788 17162 raft_consensus.cc:2781] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:34.723955 17162 raft_consensus.cc:686] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 1 LEADER]: Becoming Leader. State: Replica: 83f0dee6f68d4e59958df94f0be61966, State: Running, Role: LEADER
I20260925 14:55:34.724049 17162 consensus_queue.cc:237] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } }
I20260925 14:55:34.725023 15876 catalog_manager.cc:5495] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "83f0dee6f68d4e59958df94f0be61966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } health_report { overall_health: HEALTHY } } }
I20260925 14:55:34.725631 16637 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: NON_VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: true } } }
I20260925 14:55:34.725737 16637 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:34.726347 16636 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: attempting to promote NON_VOTER c314af0aa523494f8a3433efa62526b9 to VOTER
I20260925 14:55:34.726472 16636 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } }
I20260925 14:55:34.726852 16686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 LEARNER]: Refusing update from remote peer dcbd9ff6bcbe4f428b3db34d35134b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 14:55:34.726998 16488 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Refusing update from remote peer dcbd9ff6bcbe4f428b3db34d35134b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 14:55:34.727034 17171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 14:55:34.727296 17171 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 14:55:34.727648 16636 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, c314af0aa523494f8a3433efa62526b9 (127.9.141.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.728171 16488 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, c314af0aa523494f8a3433efa62526b9 (127.9.141.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.728641 16636 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.728745 16636 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:34.728715 16637 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.728801 16637 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:34.728654 17171 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.728966 17171 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:34.728955 16686 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, c314af0aa523494f8a3433efa62526b9 (127.9.141.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.729195 17172 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.729295 17172 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:34.729902 17163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:34.729995 17163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:34.789960 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.787073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.9.141.66:38681)
0925 14:55:34.787131 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.787133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.787157 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:55:34.787200 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.787201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.787218 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.787276 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:34.787450 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.788076 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.789004 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.789242 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.789299 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.789506 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:34.789612 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:34.789699 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.789815 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.789837 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:55:34.791872 17082 raft_consensus.cc:1261] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Refusing update from remote peer 83f0dee6f68d4e59958df94f0be61966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:34.791905 16809 raft_consensus.cc:1261] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Refusing update from remote peer 83f0dee6f68d4e59958df94f0be61966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:34.792275 17168 consensus_queue.cc:1035] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:34.792455 17162 consensus_queue.cc:1035] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:34.796562 17175 mvcc.cc:205] Tried to move back new op lower bound from 7333265960104366080 to 7333265959829340160. Current Snapshot: MvccSnapshot[applied={T|T < 7333265960104366080}]
I20260925 14:55:34.796692 17176 mvcc.cc:205] Tried to move back new op lower bound from 7333265960104366080 to 7333265959829340160. Current Snapshot: MvccSnapshot[applied={T|T < 7333265960104366080}]
I20260925 14:55:34.797936 15876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:34.798027 15876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:34.799762 15876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcbd9ff6bcbe4f428b3db34d35134b4c: Sending DeleteTablet for 3 replicas of tablet b36ba764a0534e9aab7be137090a10d1
I20260925 14:55:34.799988 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet b36ba764a0534e9aab7be137090a10d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:34 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260925 14:55:34.800076 17062 tablet_service.cc:1514] Processing DeleteTablet for tablet b36ba764a0534e9aab7be137090a10d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:34 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:37418
I20260925 14:55:34.800108 16789 tablet_service.cc:1514] Processing DeleteTablet for tablet b36ba764a0534e9aab7be137090a10d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:34 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:54460
I20260925 14:55:34.800369 17185 tablet_replica.cc:330] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: stopping tablet replica
I20260925 14:55:34.800487 17185 raft_consensus.cc:2229] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:34.800503 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet b36ba764a0534e9aab7be137090a10d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:34 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260925 14:55:34.800575 17185 raft_consensus.cc:2258] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:34.800727 17186 tablet_replica.cc:330] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: stopping tablet replica
I20260925 14:55:34.800761 17187 tablet_replica.cc:330] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: stopping tablet replica
I20260925 14:55:34.800827 17186 raft_consensus.cc:2229] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:34.800936 17186 raft_consensus.cc:2258] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:34.801070 17187 raft_consensus.cc:2229] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:34.801339 17187 raft_consensus.cc:2258] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:34.801393 17185 ts_tablet_manager.cc:1905] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:34.801512 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 15845
I20260925 14:55:34.802461 17186 ts_tablet_manager.cc:1905] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:34.802803 17187 ts_tablet_manager.cc:1905] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:34.804081 17186 ts_tablet_manager.cc:1918] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:34.804159 17186 log.cc:1196] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/b36ba764a0534e9aab7be137090a10d1
I20260925 14:55:34.804446 17186 ts_tablet_manager.cc:1939] T b36ba764a0534e9aab7be137090a10d1 P 94895020598b4f5d99ebab6732820bc7: Deleting consensus metadata
I20260925 14:55:34.805204 17187 ts_tablet_manager.cc:1918] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:34.805354 17187 log.cc:1196] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/b36ba764a0534e9aab7be137090a10d1
I20260925 14:55:34.805703 17187 ts_tablet_manager.cc:1939] T b36ba764a0534e9aab7be137090a10d1 P 1b722c1e25944aff96b7abe8ef020e95: Deleting consensus metadata
I20260925 14:55:34.806159 17185 ts_tablet_manager.cc:1918] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:34.806241 17185 log.cc:1196] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/b36ba764a0534e9aab7be137090a10d1
I20260925 14:55:34.806504 17185 ts_tablet_manager.cc:1939] T b36ba764a0534e9aab7be137090a10d1 P 83f0dee6f68d4e59958df94f0be61966: Deleting consensus metadata
W20260925 14:55:34.808662 16745 connection.cc:535] server connection from 127.0.0.1:54460 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:55:34.808692 17017 connection.cc:535] server connection from 127.0.0.1:37418 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:55:34.808868 16879 connection.cc:535] server connection from 127.0.0.1:33316 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:55:34.814953 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.800773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.9.141.125:34973)
0925 14:55:34.800811 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.800813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.806406 (+ 5593us) server_negotiation.cc:407] Connection header received
0925 14:55:34.806515 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.806516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.806531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.806563 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.807535 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.808325 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.810032 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.810350 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.810435 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.812162 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.812163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.812164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.812172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.813593 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.813993 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.813994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.813994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.814074 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.814303 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.814304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.814304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.814428 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.814540 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.814749 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.814834 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:34.818554 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.810939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.9.141.125:34973)
0925 14:55:34.810978 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.810980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.810987 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:34.811006 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.811007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.811015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.811069 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.811865 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.812641 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.813734 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.814039 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.814114 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.815652 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.815654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.815654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.815661 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.817249 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.817612 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.817613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.817613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.817673 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.817888 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.817889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.817890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.818018 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.818104 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.818363 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.818437 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:34.821768 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.800915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.9.141.124:41017)
0925 14:55:34.800956 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.800958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.815323 (+ 14365us) server_negotiation.cc:407] Connection header received
0925 14:55:34.815428 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.815430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.815442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.815502 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.816066 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.816716 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.817410 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.817742 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.817802 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.819053 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.819054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.819055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.819063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.820656 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.821000 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.821001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.821001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.821066 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.821253 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.821254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.821254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.821395 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.821507 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.821569 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.821674 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:34.831156 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:34.824679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.9.141.124:41017)
0925 14:55:34.824727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:34.824729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:34.824838 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:55:34.824883 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:34.824884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:34.824894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:34.824950 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:34.825429 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.826184 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.826901 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:34.827198 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:34.827270 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:34.828506 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:34.828507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:34.828507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:34.828514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:34.829976 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.830288 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.830289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.830290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.830347 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:34.830519 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:34.830520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:34.830521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:34.830677 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:34.830773 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:34.830989 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:55:34.831063 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:35.797117 17195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:35.796853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:60287)
0925 14:55:35.796981 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:35.797064 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
W20260925 14:55:35.797360 17129 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:36855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
I20260925 14:55:35.798933 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:35.798672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:43187)
0925 14:55:35.798817 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:35.798889 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
W20260925 14:55:35.799172 16856 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:36855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
I20260925 14:55:35.800330 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:35.800167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:41261)
0925 14:55:35.800254 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:35.800300 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":39,"threads_started":1}
W20260925 14:55:35.800546 16993 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:36855 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
I20260925 14:55:36.300164 17198 raft_consensus.cc:482] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dcbd9ff6bcbe4f428b3db34d35134b4c)
I20260925 14:55:36.300297 17198 raft_consensus.cc:504] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } }
I20260925 14:55:36.300554 17199 raft_consensus.cc:482] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dcbd9ff6bcbe4f428b3db34d35134b4c)
I20260925 14:55:36.300700 17199 raft_consensus.cc:504] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } }
I20260925 14:55:36.300779 17198 leader_election.cc:288] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855), c314af0aa523494f8a3433efa62526b9 (127.9.141.124:41017)
I20260925 14:55:36.300981 17199 leader_election.cc:288] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855), e78dcdba05fe4d0f90df66ae6e634261 (127.9.141.125:34973)
I20260925 14:55:36.301200 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.300920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46822)
0925 14:55:36.301101 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.301155 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":42,"threads_started":1}
I20260925 14:55:36.301414 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.301120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46830)
0925 14:55:36.301326 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.301383 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":53,"threads_started":1}
W20260925 14:55:36.301505 16464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
W20260925 14:55:36.301542 16662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
I20260925 14:55:36.301951 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.301847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46834)
0925 14:55:36.301890 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.301919 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:55:36.302028 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.301941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46844)
0925 14:55:36.301981 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.302002 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:55:36.302138 16662 leader_election.cc:334] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855): Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
W20260925 14:55:36.302134 16464 leader_election.cc:334] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855): Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
I20260925 14:55:36.304368 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.301064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.9.141.124:41017)
0925 14:55:36.301359 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:36.301361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:36.301372 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:36.301401 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:36.301402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:36.301414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:36.301486 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:36.301696 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.302422 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.303517 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.304055 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.304137 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.304151 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:36.304291 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 14:55:36.304324 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":37,"threads_started":1}
I20260925 14:55:36.304476 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.300962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:41017 (local address 127.0.0.1:33682)
0925 14:55:36.301162 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.301175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:36.301237 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:36.301543 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:36.301544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:36.301547 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:36.301606 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.301609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.302502 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.302503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.303425 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.303426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.304141 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.304141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.304216 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.304221 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:36.304280 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:55:36.304396 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":54,"thread_start_us":46,"threads_started":1}
I20260925 14:55:36.304728 16686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e78dcdba05fe4d0f90df66ae6e634261" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c314af0aa523494f8a3433efa62526b9" is_pre_election: true
I20260925 14:55:36.304847 16686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e78dcdba05fe4d0f90df66ae6e634261 in term 1.
I20260925 14:55:36.305001 16463 leader_election.cc:302] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c314af0aa523494f8a3433efa62526b9, e78dcdba05fe4d0f90df66ae6e634261; no voters: dcbd9ff6bcbe4f428b3db34d35134b4c
I20260925 14:55:36.305120 17198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:55:36.305158 17198 raft_consensus.cc:482] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dcbd9ff6bcbe4f428b3db34d35134b4c)
I20260925 14:55:36.305212 17198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:55:36.305227 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.301105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:34973 (local address 127.0.0.1:54834)
0925 14:55:36.301222 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.301231 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:36.301274 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:36.301981 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:36.301982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:36.301984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:36.302053 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.302055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.303034 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.303036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.304131 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.304133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.304965 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.304965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.305031 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.305036 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:36.305111 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:55:36.305153 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":44,"threads_started":1}
I20260925 14:55:36.305437 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.301385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.9.141.125:34973)
0925 14:55:36.301733 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:36.301735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:36.301743 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:36.301764 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:36.301765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:36.301884 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:36.302232 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:36.302271 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.302928 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.304201 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.304865 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.304959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.304971 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:36.305330 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:55:36.305379 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":58,"threads_started":1}
I20260925 14:55:36.305657 16488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c314af0aa523494f8a3433efa62526b9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e78dcdba05fe4d0f90df66ae6e634261" is_pre_election: true
I20260925 14:55:36.306087 17198 raft_consensus.cc:504] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } }
I20260925 14:55:36.306252 16488 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c314af0aa523494f8a3433efa62526b9 in current term 2: Already voted for candidate e78dcdba05fe4d0f90df66ae6e634261 in this term.
I20260925 14:55:36.306257 17198 leader_election.cc:288] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [CANDIDATE]: Term 2 election: Requested vote from peers dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855), c314af0aa523494f8a3433efa62526b9 (127.9.141.124:41017)
I20260925 14:55:36.306391 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.306306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46852)
0925 14:55:36.306343 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.306367 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 14:55:36.306470 16686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e78dcdba05fe4d0f90df66ae6e634261" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c314af0aa523494f8a3433efa62526b9"
I20260925 14:55:36.306538 16686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:55:36.306564 16660 leader_election.cc:302] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c314af0aa523494f8a3433efa62526b9; no voters: dcbd9ff6bcbe4f428b3db34d35134b4c, e78dcdba05fe4d0f90df66ae6e634261
I20260925 14:55:36.306747 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.306660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46856)
0925 14:55:36.306695 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.306727 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 14:55:36.306816 16464 leader_election.cc:334] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855): Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
I20260925 14:55:36.307256 16686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e78dcdba05fe4d0f90df66ae6e634261 in term 2.
I20260925 14:55:36.307334 17199 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:55:36.307423 16463 leader_election.cc:302] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c314af0aa523494f8a3433efa62526b9, e78dcdba05fe4d0f90df66ae6e634261; no voters: dcbd9ff6bcbe4f428b3db34d35134b4c
I20260925 14:55:36.307549 17198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:55:36.307686 17198 raft_consensus.cc:686] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [term 2 LEADER]: Becoming Leader. State: Replica: e78dcdba05fe4d0f90df66ae6e634261, State: Running, Role: LEADER
I20260925 14:55:36.307773 17198 consensus_queue.cc:237] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } }
I20260925 14:55:36.308130 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e78dcdba05fe4d0f90df66ae6e634261. Latest consensus state: current_term: 2 leader_uuid: "e78dcdba05fe4d0f90df66ae6e634261" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:36.308245 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:36.308471 17210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:36.308856 17210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:36.308920 17210 catalog_manager.cc:1259] Loaded cluster ID: 8daf7cc897dc4e01afca6554a3eb2ccf
I20260925 14:55:36.308972 17210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:36.309131 17210 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:36.309227 17210 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Loaded TSK: 0
I20260925 14:55:36.309378 17210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:36.314215 16477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:36.315825 16477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:36.316932 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.316810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46878)
0925 14:55:36.316857 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.316886 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:55:36.317030 16686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [term 2 FOLLOWER]: Refusing update from remote peer e78dcdba05fe4d0f90df66ae6e634261: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 14:55:36.317273 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.317167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.0.0.1:46882)
0925 14:55:36.317221 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.317251 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:55:36.317282 17198 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 14:55:36.317425 16464 consensus_peers.cc:487] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 -> Peer dcbd9ff6bcbe4f428b3db34d35134b4c (127.9.141.126:36855): Couldn't send request to peer dcbd9ff6bcbe4f428b3db34d35134b4c. Status: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:55:36.317945 17199 sys_catalog.cc:455] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e78dcdba05fe4d0f90df66ae6e634261. Latest consensus state: current_term: 2 leader_uuid: "e78dcdba05fe4d0f90df66ae6e634261" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:36.318042 17199 sys_catalog.cc:458] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:36.318437 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e78dcdba05fe4d0f90df66ae6e634261" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:36.318517 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:36.318565 17199 sys_catalog.cc:455] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e78dcdba05fe4d0f90df66ae6e634261" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "dcbd9ff6bcbe4f428b3db34d35134b4c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36855 } } peers { permanent_uuid: "e78dcdba05fe4d0f90df66ae6e634261" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 34973 } attrs { promote: false } } peers { permanent_uuid: "c314af0aa523494f8a3433efa62526b9" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 41017 } attrs { promote: false } } }
I20260925 14:55:36.318641 17199 sys_catalog.cc:458] T 00000000000000000000000000000000 P c314af0aa523494f8a3433efa62526b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:55:36.323731 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.320448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:41787 (local address 127.0.0.1:57764)
0925 14:55:36.320482 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.320492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:36.320538 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:36.321046 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:36.321047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:36.321050 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:36.321091 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.321093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.321998 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.321999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.322820 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.322823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.323505 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.323506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.323569 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.323572 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:36.323627 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:55:36.323666 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 14:55:36.323753 17218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.320625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.9.141.65:41787)
0925 14:55:36.320942 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:36.320944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:36.320951 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:36.320976 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:36.320976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:36.320985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:36.321045 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:36.321167 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.321899 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.322915 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.323395 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.323492 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.323504 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:36.323643 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 14:55:36.323684 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":60,"threads_started":1}
I20260925 14:55:36.324061 16789 tablet_service.cc:1467] Processing CreateTablet for tablet 78f46795c18e42778f50a5eec5b7c878 (DEFAULT_TABLE table=test-table [id=a01974ffbe9342ccaed5aafa5f3cb5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:36.324224 16789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f46795c18e42778f50a5eec5b7c878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:36.324232 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.320475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:44533 (local address 127.0.0.1:41316)
0925 14:55:36.320650 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.320660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:36.320698 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:36.320976 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:36.320978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:36.320980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:36.321061 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.321062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.321964 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.321964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.323111 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.323112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.323949 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.323950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.324036 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.324039 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:36.324114 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:55:36.324154 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":145,"threads_started":1}
I20260925 14:55:36.324226 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.320730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.9.141.67:44533)
0925 14:55:36.320824 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:36.320826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:36.320834 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:36.320860 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:36.320861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:36.320871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:36.320966 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:36.321121 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.321887 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.323278 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.323851 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.323927 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.323942 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:36.324123 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:55:36.324161 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
I20260925 14:55:36.324505 17062 tablet_service.cc:1467] Processing CreateTablet for tablet 78f46795c18e42778f50a5eec5b7c878 (DEFAULT_TABLE table=test-table [id=a01974ffbe9342ccaed5aafa5f3cb5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:36.324644 17062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f46795c18e42778f50a5eec5b7c878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:36.325512 17221 tablet_bootstrap.cc:492] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95: Bootstrap starting.
I20260925 14:55:36.325887 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.320382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:38681 (local address 127.0.0.1:57350)
0925 14:55:36.320436 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.320446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:36.320486 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:36.321095 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:36.321096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:36.321098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:36.321137 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.321138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.321846 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.321847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.323045 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:36.323048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.325665 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.325666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:36.325733 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.325737 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:36.325767 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 14:55:36.325811 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":33}
I20260925 14:55:36.325932 17221 tablet_bootstrap.cc:654] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:36.326126 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.320826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.9.141.66:38681)
0925 14:55:36.320952 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:36.320954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:36.320969 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:55:36.320992 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:36.320993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:36.321004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:36.321194 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:36.321230 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.321767 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.323215 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.325577 (+ 2362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.325997 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.326013 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:36.326031 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 14:55:36.326080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260925 14:55:36.326360 16926 tablet_service.cc:1467] Processing CreateTablet for tablet 78f46795c18e42778f50a5eec5b7c878 (DEFAULT_TABLE table=test-table [id=a01974ffbe9342ccaed5aafa5f3cb5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:36.326478 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78f46795c18e42778f50a5eec5b7c878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:36.326560 17222 tablet_bootstrap.cc:492] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Bootstrap starting.
I20260925 14:55:36.326808 17221 tablet_bootstrap.cc:492] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95: No bootstrap required, opened a new log
I20260925 14:55:36.326860 17221 ts_tablet_manager.cc:1397] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:36.326939 17222 tablet_bootstrap.cc:654] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:36.326996 17221 raft_consensus.cc:348] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.327072 17221 raft_consensus.cc:374] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:36.327100 17221 raft_consensus.cc:729] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b722c1e25944aff96b7abe8ef020e95, State: Initialized, Role: FOLLOWER
I20260925 14:55:36.327142 17221 consensus_queue.cc:260] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.327309 17221 ts_tablet_manager.cc:1428] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:36.327459 17222 tablet_bootstrap.cc:492] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: No bootstrap required, opened a new log
I20260925 14:55:36.327526 17222 ts_tablet_manager.cc:1397] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:36.327661 17222 raft_consensus.cc:348] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.327741 17222 raft_consensus.cc:374] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:36.327769 17222 raft_consensus.cc:729] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94895020598b4f5d99ebab6732820bc7, State: Initialized, Role: FOLLOWER
I20260925 14:55:36.327816 17222 consensus_queue.cc:260] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.327905 17226 tablet_bootstrap.cc:492] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Bootstrap starting.
I20260925 14:55:36.327960 17222 ts_tablet_manager.cc:1428] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:36.328248 17226 tablet_bootstrap.cc:654] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:36.328581 16857 heartbeater.cc:503] Master 127.9.141.125:34973 was elected leader, sending a full tablet report...
I20260925 14:55:36.328948 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.328239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:33545)
0925 14:55:36.328868 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.328911 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":76,"threads_started":1}
I20260925 14:55:36.329006 17130 heartbeater.cc:503] Master 127.9.141.125:34973 was elected leader, sending a full tablet report...
I20260925 14:55:36.329166 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.328095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:40245)
0925 14:55:36.329091 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.329128 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":72,"threads_started":1}
I20260925 14:55:36.329473 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.329303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:56107)
0925 14:55:36.329419 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.329445 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 14:55:36.329562 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.329360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:50547)
0925 14:55:36.329488 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.329521 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":94,"threads_started":1}
I20260925 14:55:36.329917 17226 tablet_bootstrap.cc:492] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: No bootstrap required, opened a new log
I20260925 14:55:36.329958 17226 ts_tablet_manager.cc:1397] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:36.330049 17226 raft_consensus.cc:348] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.330096 17226 raft_consensus.cc:374] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:36.330109 17226 raft_consensus.cc:729] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f0dee6f68d4e59958df94f0be61966, State: Initialized, Role: FOLLOWER
I20260925 14:55:36.330168 17226 consensus_queue.cc:260] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.330294 17226 ts_tablet_manager.cc:1428] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:36.330757 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.330568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:48261)
0925 14:55:36.330686 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.330727 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":77,"threads_started":1}
I20260925 14:55:36.340456 17233 raft_consensus.cc:482] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:36.340552 17233 raft_consensus.cc:504] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.340692 17233 leader_election.cc:288] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787), 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533)
I20260925 14:55:36.340864 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f46795c18e42778f50a5eec5b7c878" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94895020598b4f5d99ebab6732820bc7" is_pre_election: true
I20260925 14:55:36.340939 17082 raft_consensus.cc:2454] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 0.
I20260925 14:55:36.341084 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f46795c18e42778f50a5eec5b7c878" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b722c1e25944aff96b7abe8ef020e95" is_pre_election: true
I20260925 14:55:36.341133 16881 leader_election.cc:302] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f0dee6f68d4e59958df94f0be61966, 94895020598b4f5d99ebab6732820bc7; no voters:
I20260925 14:55:36.341171 16809 raft_consensus.cc:2454] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 0.
I20260925 14:55:36.341215 17233 raft_consensus.cc:2781] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:36.341248 17233 raft_consensus.cc:482] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:36.341266 17233 raft_consensus.cc:3037] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:36.342125 17233 raft_consensus.cc:504] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.342254 17233 leader_election.cc:288] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 election: Requested vote from peers 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787), 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533)
I20260925 14:55:36.342393 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f46795c18e42778f50a5eec5b7c878" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94895020598b4f5d99ebab6732820bc7"
I20260925 14:55:36.342466 17082 raft_consensus.cc:3037] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:36.342986 17082 raft_consensus.cc:2454] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 1.
I20260925 14:55:36.343106 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f46795c18e42778f50a5eec5b7c878" candidate_uuid: "83f0dee6f68d4e59958df94f0be61966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b722c1e25944aff96b7abe8ef020e95"
I20260925 14:55:36.343179 16809 raft_consensus.cc:3037] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:36.343165 16881 leader_election.cc:302] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f0dee6f68d4e59958df94f0be61966, 94895020598b4f5d99ebab6732820bc7; no voters:
I20260925 14:55:36.343724 16809 raft_consensus.cc:2454] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f0dee6f68d4e59958df94f0be61966 in term 1.
I20260925 14:55:36.343842 17233 raft_consensus.cc:2781] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:36.344026 17233 raft_consensus.cc:686] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 1 LEADER]: Becoming Leader. State: Replica: 83f0dee6f68d4e59958df94f0be61966, State: Running, Role: LEADER
I20260925 14:55:36.344101 16994 heartbeater.cc:503] Master 127.9.141.125:34973 was elected leader, sending a full tablet report...
I20260925 14:55:36.344086 17233 consensus_queue.cc:237] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } }
I20260925 14:55:36.344465 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.344313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:40079)
0925 14:55:36.344371 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.344435 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:55:36.344655 16477 catalog_manager.cc:5495] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "83f0dee6f68d4e59958df94f0be61966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f0dee6f68d4e59958df94f0be61966" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:36.351950 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.349989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.9.141.66:38681)
0925 14:55:36.350055 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:36.350057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:36.350065 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:36.350086 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:36.350089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:36.350102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:36.350147 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:36.350255 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.350716 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.351400 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:36.351575 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:36.351622 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:36.351746 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:36.351769 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:36.351806 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:36.351858 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:55:36.351872 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:55:36.352664 17082 raft_consensus.cc:1261] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Refusing update from remote peer 83f0dee6f68d4e59958df94f0be61966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:36.352715 16809 raft_consensus.cc:1261] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Refusing update from remote peer 83f0dee6f68d4e59958df94f0be61966: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:36.352924 17233 consensus_queue.cc:1035] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b722c1e25944aff96b7abe8ef020e95" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 41787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:36.353055 17235 consensus_queue.cc:1035] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94895020598b4f5d99ebab6732820bc7" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:36.353731 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.353632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:56327)
0925 14:55:36.353665 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.353699 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 14:55:36.353765 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.353666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:53285)
0925 14:55:36.353703 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.353729 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 14:55:36.354758 17238 mvcc.cc:205] Tried to move back new op lower bound from 7333265966499028992 to 7333265966465814528. Current Snapshot: MvccSnapshot[applied={T|T < 7333265966499028992}]
I20260925 14:55:36.355029 17229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.354937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.67:60519)
0925 14:55:36.354971 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.355001 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 14:55:36.355496 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.355393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:34809)
0925 14:55:36.355428 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.355463 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 14:55:36.355873 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.355766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.66:44593)
0925 14:55:36.355821 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.355846 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:55:36.356098 16477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:36.356172 16477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:36.356617 17232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.356525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:36855 (local address 127.9.141.65:40025)
0925 14:55:36.356562 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.356587 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:36855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:55:36.357431 16477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e78dcdba05fe4d0f90df66ae6e634261: Sending DeleteTablet for 3 replicas of tablet 78f46795c18e42778f50a5eec5b7c878
I20260925 14:55:36.357628 16926 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f46795c18e42778f50a5eec5b7c878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:36 UTC) from {username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:57350
I20260925 14:55:36.357685 17062 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f46795c18e42778f50a5eec5b7c878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:36 UTC) from {username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:41316
I20260925 14:55:36.357628 16925 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f46795c18e42778f50a5eec5b7c878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:36 UTC) from {username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:57350
I20260925 14:55:36.357900 17249 tablet_replica.cc:330] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: stopping tablet replica
I20260925 14:55:36.357941 16789 tablet_service.cc:1514] Processing DeleteTablet for tablet 78f46795c18e42778f50a5eec5b7c878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:36 UTC) from {username='oryx', principal='oryx/127.9.141.125@KRBTEST.COM'} at 127.0.0.1:57764
I20260925 14:55:36.357966 17249 raft_consensus.cc:2229] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:36.358008 17249 raft_consensus.cc:2258] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:36.358132 17250 tablet_replica.cc:330] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95: stopping tablet replica
I20260925 14:55:36.358181 17250 raft_consensus.cc:2229] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:36.358178 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16635
I20260925 14:55:36.358222 17250 raft_consensus.cc:2258] T 78f46795c18e42778f50a5eec5b7c878 P 1b722c1e25944aff96b7abe8ef020e95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:36.359078 17248 tablet_replica.cc:330] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: stopping tablet replica
I20260925 14:55:36.359133 17248 raft_consensus.cc:2229] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:36.359189 17248 raft_consensus.cc:2258] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:36.359476 17248 ts_tablet_manager.cc:1905] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:36.359519 17249 ts_tablet_manager.cc:1905] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:36.361272 17249 ts_tablet_manager.cc:1918] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:36.361272 17248 ts_tablet_manager.cc:1918] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:36.361334 17249 log.cc:1196] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/78f46795c18e42778f50a5eec5b7c878
I20260925 14:55:36.361334 17248 log.cc:1196] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/78f46795c18e42778f50a5eec5b7c878
I20260925 14:55:36.361757 17248 ts_tablet_manager.cc:1939] T 78f46795c18e42778f50a5eec5b7c878 P 83f0dee6f68d4e59958df94f0be61966: Deleting consensus metadata
I20260925 14:55:36.361757 17249 ts_tablet_manager.cc:1939] T 78f46795c18e42778f50a5eec5b7c878 P 94895020598b4f5d99ebab6732820bc7: Deleting consensus metadata
I20260925 14:55:36.362171 16464 catalog_manager.cc:4841] TS 94895020598b4f5d99ebab6732820bc7 (127.9.141.67:44533): tablet 78f46795c18e42778f50a5eec5b7c878 (table test-table [id=a01974ffbe9342ccaed5aafa5f3cb5f7]) successfully deleted
I20260925 14:55:36.362214 16463 catalog_manager.cc:4841] TS 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66:38681): tablet 78f46795c18e42778f50a5eec5b7c878 (table test-table [id=a01974ffbe9342ccaed5aafa5f3cb5f7]) successfully deleted
W20260925 14:55:36.362355 16463 catalog_manager.cc:4808] TS 83f0dee6f68d4e59958df94f0be61966 (127.9.141.66:38681): delete failed for tablet 78f46795c18e42778f50a5eec5b7c878 because the tablet was not found. No further retry: Not found: Tablet not found: 78f46795c18e42778f50a5eec5b7c878
I20260925 14:55:36.364239 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16863
I20260925 14:55:36.364296 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:36.364183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:41787 (local address 127.0.0.1:57774)
0925 14:55:36.364221 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:36.364246 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:41787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 14:55:36.364449 16462 catalog_manager.cc:4584] TS 1b722c1e25944aff96b7abe8ef020e95 (127.9.141.65:41787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78f46795c18e42778f50a5eec5b7c878: Network error: Client connection negotiation failed: client connection to 127.9.141.65:41787: connect: Connection refused (error 111)
I20260925 14:55:36.370360 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16999
I20260925 14:55:36.375171 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16341
I20260925 14:55:36.379833 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 16537
2026-09-25T14:55:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6206 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:36 dist-test-slave-1sv7 krb5kdc[17253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:36 dist-test-slave-1sv7 krb5kdc[17253](info): set up 2 sockets
Sep 25 14:55:36 dist-test-slave-1sv7 krb5kdc[17253](info): commencing operation
krb5kdc: starting...
W20260925 14:55:37.464911 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:37 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab.
Entry for principal oryx/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab.
I20260925 14:55:37.592754 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:34219
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab
--principal=oryx/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:37.679970 17266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:37.680171 17266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:37.680194 17266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:37.681295 17266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:37.681330 17266 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:55:37.681344 17266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:37.681356 17266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:37.681370 17266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:37.681380 17266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:37.682786 17266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:34219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.126.keytab
--principal=oryx/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:37.682982 17266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:37.683142 17266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:37.688068 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:37.688266 17266 server_base.cc:1034] running on GCE node
W20260925 14:55:37.688035 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:37.688052 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:37.688679 17266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:37.688915 17266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:37.690073 17266 hybrid_clock.cc:648] HybridClock initialized: now 1790348137690060 us; error 28 us; skew 500 ppm
Sep 25 14:55:37 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348137, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:37.698443 17266 init.cc:377] Logged in from keytab as oryx/127.9.141.126@KRBTEST.COM (short username oryx)
I20260925 14:55:37.698966 17266 webserver.cc:460] Webserver started at http://127.9.141.126:44393/ using document root <none> and password file <none>
I20260925 14:55:37.699203 17266 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:37.699272 17266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:37.699424 17266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:37.700573 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a"
format_stamp: "Formatted at 2026-09-25 14:55:37 on dist-test-slave-1sv7"
server_key: "7c6cd25850d49ad785ab47bde78691f1"
server_key_iv: "efd313a71b1920ee552e302e49b73481"
server_key_version: "encryptionkey@0"
I20260925 14:55:37.700943 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a"
format_stamp: "Formatted at 2026-09-25 14:55:37 on dist-test-slave-1sv7"
server_key: "7c6cd25850d49ad785ab47bde78691f1"
server_key_iv: "efd313a71b1920ee552e302e49b73481"
server_key_version: "encryptionkey@0"
I20260925 14:55:37.702373 17266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:55:37.703194 17281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:37.703418 17266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:37.703523 17266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a"
format_stamp: "Formatted at 2026-09-25 14:55:37 on dist-test-slave-1sv7"
server_key: "7c6cd25850d49ad785ab47bde78691f1"
server_key_iv: "efd313a71b1920ee552e302e49b73481"
server_key_version: "encryptionkey@0"
I20260925 14:55:37.703622 17266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:37.731050 17266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:37.733526 17266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:37.733718 17266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:37.738950 17266 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:34219
I20260925 14:55:37.738997 17343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:34219 every 8 connection(s)
I20260925 14:55:37.739362 17266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:37.740046 17344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:37.742460 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Bootstrap starting.
I20260925 14:55:37.743053 17344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:37.743311 17344 log.cc:824] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:37.743999 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: No bootstrap required, opened a new log
I20260925 14:55:37.746206 17344 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 34219 } }
I20260925 14:55:37.746362 17344 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:37.746388 17344 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7fe0fdef3154ca2bb4a4cdeefc9368a, State: Initialized, Role: FOLLOWER
I20260925 14:55:37.746495 17344 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 34219 } }
I20260925 14:55:37.746551 17344 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:37.746613 17344 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:37.746672 17344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:37.747243 17344 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 34219 } }
I20260925 14:55:37.747332 17344 leader_election.cc:302] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7fe0fdef3154ca2bb4a4cdeefc9368a; no voters:
I20260925 14:55:37.747531 17344 leader_election.cc:288] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:37.747639 17347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:37.747794 17347 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [term 1 LEADER]: Becoming Leader. State: Replica: a7fe0fdef3154ca2bb4a4cdeefc9368a, State: Running, Role: LEADER
I20260925 14:55:37.747788 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 17266
I20260925 14:55:37.747898 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:37.747968 17347 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 34219 } }
I20260925 14:55:37.747812 17344 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:37.748121 9781 external_mini_cluster.cc:1428] Setting key 5646f8727afeb0fdaf816d97cdacbbdb
I20260925 14:55:37.749238 17349 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 34219 } } }
I20260925 14:55:37.749320 17349 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:37.749541 17364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:37.749625 17348 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7fe0fdef3154ca2bb4a4cdeefc9368a. Latest consensus state: current_term: 1 leader_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7fe0fdef3154ca2bb4a4cdeefc9368a" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 34219 } } }
I20260925 14:55:37.749953 17348 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:37.750389 17364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:37.751765 17364 catalog_manager.cc:1347] Generated new cluster ID: 99e57cc1afb84d84ace285ff782a1c2e
I20260925 14:55:37.751813 17364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:37 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:37.760429 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.748693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.9.141.126:34219)
0925 14:55:37.749085 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:37.749096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:37.749132 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:55:37.749258 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:37.749265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:37.749320 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:37.750096 (+ 776us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:37.750314 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.751870 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.752930 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.753615 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.753749 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:37.756744 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:37.756748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:37.756748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:37.756765 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:37.758750 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:37.759183 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:37.759186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:37.759186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:37.759281 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:37.759506 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:37.759507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:37.759507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:37.759792 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:37.759942 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:37.760075 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 14:55:37.760203 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":76,"threads_started":1}
I20260925 14:55:37.766441 17364 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:37.767222 17364 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:37.775520 17364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Generated new TSK 0
I20260925 14:55:37.775707 17364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab.
Entry for principal oryx/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab.
I20260925 14:55:37.808722 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:34219
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:37.897292 17370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:37.897500 17370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:37.897524 17370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:37.898691 17370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:37.898736 17370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:37.898757 17370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:37.898774 17370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:37.900297 17370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:34219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:37.900588 17370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:37.900815 17370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:37.906435 17375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:37.906446 17378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:37.906575 17376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:37.906455 17370 server_base.cc:1034] running on GCE node
I20260925 14:55:37.906864 17370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:37.907105 17370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:37.908272 17370 hybrid_clock.cc:648] HybridClock initialized: now 1790348137908254 us; error 30 us; skew 500 ppm
Sep 25 14:55:37 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348137, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:37.910931 17370 init.cc:377] Logged in from keytab as oryx/127.9.141.65@KRBTEST.COM (short username oryx)
I20260925 14:55:37.911448 17370 webserver.cc:460] Webserver started at http://127.9.141.65:43829/ using document root <none> and password file <none>
I20260925 14:55:37.911684 17370 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:37.911753 17370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:37.911903 17370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:37.913003 17370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "e99acd84aace4b078e6f751d87d3ca18"
format_stamp: "Formatted at 2026-09-25 14:55:37 on dist-test-slave-1sv7"
server_key: "e7544e07ee606a2b9be2849ef4af1f2c"
server_key_iv: "258b7db7f55956363f45136a7929768b"
server_key_version: "encryptionkey@0"
I20260925 14:55:37.913626 17370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "e99acd84aace4b078e6f751d87d3ca18"
format_stamp: "Formatted at 2026-09-25 14:55:37 on dist-test-slave-1sv7"
server_key: "e7544e07ee606a2b9be2849ef4af1f2c"
server_key_iv: "258b7db7f55956363f45136a7929768b"
server_key_version: "encryptionkey@0"
I20260925 14:55:37.915153 17370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:37.916029 17385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:37.916293 17370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:37.916452 17370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "e99acd84aace4b078e6f751d87d3ca18"
format_stamp: "Formatted at 2026-09-25 14:55:37 on dist-test-slave-1sv7"
server_key: "e7544e07ee606a2b9be2849ef4af1f2c"
server_key_iv: "258b7db7f55956363f45136a7929768b"
server_key_version: "encryptionkey@0"
I20260925 14:55:37.916548 17370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:37.929170 17370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:37.931057 17370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:37.931238 17370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:37.931816 17370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:37.931878 17370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:37.931965 17370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:37.932021 17370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:55:37 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348137, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:37.940140 17370 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:45437
I20260925 14:55:37.940224 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:45437 every 8 connection(s)
I20260925 14:55:37.940686 17370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:37.942339 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.932197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:41257 (local address 127.9.141.126:34219)
0925 14:55:37.932249 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:37.932251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:37.932571 (+ 320us) server_negotiation.cc:407] Connection header received
0925 14:55:37.933025 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:37.933027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:37.933044 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:37.933134 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:37.933725 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.934301 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.935438 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.935661 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.935748 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:37.939074 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:37.939076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:37.939076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:37.939084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:37.940909 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:37.941454 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:37.941455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:37.941455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:37.941513 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:37.941775 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:37.941776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:37.941776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:37.941887 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:37.941941 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:37.942202 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:55:37.942270 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:55:37.942559 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.932239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.9.141.65:41257)
0925 14:55:37.932433 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:37.932472 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:37.932930 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:37.933195 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:37.933198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:37.933421 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:37.933591 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:37.933595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.934397 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.934398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:37.935349 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:37.935351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.935758 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.935759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:37.935882 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:37.936454 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:37.936466 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:37.938962 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:37.941023 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:37.941030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:37.941032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:37.941332 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:37.941591 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:37.941592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:37.941592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:37.941692 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:37.941948 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:37.941953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:37.942083 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:55:37.942173 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:55:37.942302 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260925 14:55:37.943142 17501 heartbeater.cc:347] Connected to a master server at 127.9.141.126:34219
I20260925 14:55:37.943245 17501 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:37.943434 17501 heartbeater.cc:511] Master 127.9.141.126:34219 requested a full tablet report, sending...
I20260925 14:55:37.944077 17297 ts_manager.cc:194] Registered new tserver with Master: e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437)
I20260925 14:55:37.944139 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 17370
I20260925 14:55:37.944275 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:37.944554 9781 external_mini_cluster.cc:1428] Setting key cd7e642dc44a4001b1c8aeb4de853506
I20260925 14:55:37.944968 17297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:41257
I20260925 14:55:37.951784 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.944216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.9.141.65:38183)
0925 14:55:37.944396 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:37.944409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:37.944462 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:37.945335 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:37.945337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:37.945539 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:37.945616 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:37.945618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.946322 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.946323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:37.947292 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:37.947294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.947788 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.947789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:37.947840 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:37.948423 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:37.948428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:37.948810 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:37.950710 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:37.950711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:37.950712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:37.950899 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:37.951116 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:37.951117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:37.951117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:37.951170 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:37.951513 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:37.951514 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:37.951556 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:55:37.951624 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:55:37.951709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260925 14:55:37.951860 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:37.945168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:38183 (local address 127.9.141.126:34219)
0925 14:55:37.945219 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:37.945220 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:37.945228 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:37.945247 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:37.945248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:37.945258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:37.945313 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:37.945696 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.946216 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.947382 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:37.947685 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:37.947763 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:37.948889 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:37.948891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:37.948891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:37.948898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:37.950619 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:37.950979 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:37.950980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:37.950980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:37.951043 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:37.951238 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:37.951239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:37.951239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:37.951393 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:37.951470 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:37.951674 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:55:37.951759 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for oryx/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab.
Entry for principal oryx/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab.
I20260925 14:55:37.972713 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:34219
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:38.054418 17510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:38.054625 17510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:38.054649 17510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:38.055739 17510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:38.055773 17510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:38.055794 17510 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:38.055810 17510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:38.057304 17510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:34219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:38.057497 17510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:38.057665 17510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:38.062634 17518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:38.062635 17515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:38.062770 17516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:38.062994 17510 server_base.cc:1034] running on GCE node
I20260925 14:55:38.063262 17510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:38.063529 17510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:38.064689 17510 hybrid_clock.cc:648] HybridClock initialized: now 1790348138064686 us; error 51 us; skew 500 ppm
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:38.067171 17510 init.cc:377] Logged in from keytab as oryx/127.9.141.66@KRBTEST.COM (short username oryx)
I20260925 14:55:38.067687 17510 webserver.cc:460] Webserver started at http://127.9.141.66:39703/ using document root <none> and password file <none>
I20260925 14:55:38.067910 17510 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:38.067977 17510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:38.068126 17510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:38.069242 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
format_stamp: "Formatted at 2026-09-25 14:55:38 on dist-test-slave-1sv7"
server_key: "5c2e84e2c9b8b7c1dc789ab3efa38b6b"
server_key_iv: "96f0622b16b0f0eb11e3b3a6a3f3afa3"
server_key_version: "encryptionkey@0"
I20260925 14:55:38.069622 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
format_stamp: "Formatted at 2026-09-25 14:55:38 on dist-test-slave-1sv7"
server_key: "5c2e84e2c9b8b7c1dc789ab3efa38b6b"
server_key_iv: "96f0622b16b0f0eb11e3b3a6a3f3afa3"
server_key_version: "encryptionkey@0"
I20260925 14:55:38.070973 17510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:38.071768 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.071954 17510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:38.072052 17510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
format_stamp: "Formatted at 2026-09-25 14:55:38 on dist-test-slave-1sv7"
server_key: "5c2e84e2c9b8b7c1dc789ab3efa38b6b"
server_key_iv: "96f0622b16b0f0eb11e3b3a6a3f3afa3"
server_key_version: "encryptionkey@0"
I20260925 14:55:38.072139 17510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:38.090348 17510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:38.092471 17510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:38.092631 17510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:38.093230 17510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:38.093277 17510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.093312 17510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:38.093341 17510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:38.100832 17510 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:32775
I20260925 14:55:38.100917 17640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:32775 every 8 connection(s)
I20260925 14:55:38.101264 17510 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:38.103801 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.093654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:59473 (local address 127.9.141.126:34219)
0925 14:55:38.093727 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.093730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.094014 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:55:38.094734 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.094735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.094749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.094830 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.095802 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.096614 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.098264 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.098490 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.098574 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.100847 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:38.100849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:38.100849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:38.100858 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:38.102375 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.102805 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.102806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.102806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.102876 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.103137 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.103138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.103139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.103265 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:38.103336 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.103625 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.103713 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:55:38.103978 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.093641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.9.141.66:59473)
0925 14:55:38.093858 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.093910 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.094609 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.094966 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.094971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.095384 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:38.095688 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.095697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.096690 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.096691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.098162 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.098166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.098600 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.098601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.098727 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.099228 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:38.099236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:38.100742 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:38.102480 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.102487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.102489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.102710 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.102946 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.102947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.102948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.103061 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.103346 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:38.103353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:38.103487 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.103590 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.103746 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260925 14:55:38.104490 17641 heartbeater.cc:347] Connected to a master server at 127.9.141.126:34219
I20260925 14:55:38.104583 17641 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:38.104815 17641 heartbeater.cc:511] Master 127.9.141.126:34219 requested a full tablet report, sending...
I20260925 14:55:38.105250 17298 ts_manager.cc:194] Registered new tserver with Master: 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775)
I20260925 14:55:38.105783 17298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:59473
I20260925 14:55:38.107553 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 17510
I20260925 14:55:38.107636 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:38.107884 9781 external_mini_cluster.cc:1428] Setting key 7604aec8e3929debf652b099c589a141
I20260925 14:55:38.111824 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.105504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.9.141.66:59349)
0925 14:55:38.105686 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.105703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.105757 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.105889 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.105890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.106123 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:38.106189 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.106191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.106980 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.106981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.107761 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.107764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.108216 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.108217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.108293 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.108752 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:38.108756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:38.109051 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:38.110665 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.110667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.110667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.110918 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.111110 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.111111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.111111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.111187 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.111530 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:38.111531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:38.111569 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.111629 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.111748 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260925 14:55:38.111879 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.105599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:59349 (local address 127.9.141.126:34219)
0925 14:55:38.105638 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.105640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.105763 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:55:38.105792 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.105793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.105804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.105875 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.106337 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.106866 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.107883 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.108133 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.108216 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.109142 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:38.109143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:38.109143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:38.109151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:38.110577 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.111003 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.111004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.111004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.111050 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.111264 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.111265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.111265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.111459 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:38.111531 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.111696 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.111778 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for oryx/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab.
Entry for principal oryx/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab.
I20260925 14:55:38.136303 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:34219
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:38.215992 17650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:38.216199 17650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:38.216223 17650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:38.217406 17650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:38.217446 17650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:38.217466 17650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:55:38.217484 17650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:38.219044 17650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.141.126:34219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:38.219280 17650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:38.219503 17650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:38.224671 17656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:38.224782 17655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:38.224671 17658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:38.224858 17650 server_base.cc:1034] running on GCE node
I20260925 14:55:38.225102 17650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:38.225358 17650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:38.226541 17650 hybrid_clock.cc:648] HybridClock initialized: now 1790348138226518 us; error 43 us; skew 500 ppm
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:38.229280 17650 init.cc:377] Logged in from keytab as oryx/127.9.141.67@KRBTEST.COM (short username oryx)
I20260925 14:55:38.229795 17650 webserver.cc:460] Webserver started at http://127.9.141.67:36671/ using document root <none> and password file <none>
I20260925 14:55:38.230047 17650 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:38.230116 17650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:38.230268 17650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:38.231287 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21"
format_stamp: "Formatted at 2026-09-25 14:55:38 on dist-test-slave-1sv7"
server_key: "ccded790073a3a40e7e412f08adf3a50"
server_key_iv: "55220038daea078f639f12d6c51574c6"
server_key_version: "encryptionkey@0"
I20260925 14:55:38.231658 17650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21"
format_stamp: "Formatted at 2026-09-25 14:55:38 on dist-test-slave-1sv7"
server_key: "ccded790073a3a40e7e412f08adf3a50"
server_key_iv: "55220038daea078f639f12d6c51574c6"
server_key_version: "encryptionkey@0"
I20260925 14:55:38.233057 17650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:38.233839 17665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.234045 17650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:38.234140 17650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21"
format_stamp: "Formatted at 2026-09-25 14:55:38 on dist-test-slave-1sv7"
server_key: "ccded790073a3a40e7e412f08adf3a50"
server_key_iv: "55220038daea078f639f12d6c51574c6"
server_key_version: "encryptionkey@0"
I20260925 14:55:38.234222 17650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:38.265033 17650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:38.267108 17650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:38.267274 17650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:38.267894 17650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:38.267985 17650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.268030 17650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:38.268044 17650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:38.275367 17650 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39861
I20260925 14:55:38.275740 17780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39861 every 8 connection(s)
I20260925 14:55:38.276095 17650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:38.278350 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.268328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:37127 (local address 127.9.141.126:34219)
0925 14:55:38.268432 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.268434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.268701 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:55:38.269239 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.269240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.269254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.269334 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.269990 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.270501 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.271663 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.271879 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.271967 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.275182 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:38.275183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:38.275183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:38.275191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:38.276891 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.277371 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.277372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.277372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.277435 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.277697 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.277698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.277699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.277818 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:38.277896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.278172 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.278250 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 14:55:38.278522 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.268397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.9.141.67:37127)
0925 14:55:38.268561 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.268602 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.269137 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.269413 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.269417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.269651 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:38.269891 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.269895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.270608 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.270610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.271567 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.271570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.271979 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.271980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.272102 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.272655 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:38.272665 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:38.275070 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:38.276992 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.276998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.277001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.277273 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.277502 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.277503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.277503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.277620 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.277886 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:38.277892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:38.278032 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.278137 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.278295 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260925 14:55:38.278983 17781 heartbeater.cc:347] Connected to a master server at 127.9.141.126:34219
I20260925 14:55:38.279065 17781 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:38.279250 17781 heartbeater.cc:511] Master 127.9.141.126:34219 requested a full tablet report, sending...
I20260925 14:55:38.279610 17298 ts_manager.cc:194] Registered new tserver with Master: 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:38.280056 17298 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:37127
I20260925 14:55:38.281452 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 17650
I20260925 14:55:38.281553 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:38.281738 9781 external_mini_cluster.cc:1428] Setting key e6f4fdba2d10106acdce38daa0f5107a
I20260925 14:55:38.282632 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:38.286243 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.280013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.9.141.67:41317)
0925 14:55:38.280155 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.280167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.280227 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.280358 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.280359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.280584 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:38.280650 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.280652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.281314 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.281315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.282108 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.282116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.282539 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.282540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.282617 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.283071 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:38.283076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:38.283450 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:38.285252 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.285253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.285254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.285430 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.285627 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.285627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.285627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.285683 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.285971 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:38.285972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:38.286012 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.286082 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.286161 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260925 14:55:38.286345 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.280155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:41317 (local address 127.9.141.126:34219)
0925 14:55:38.280259 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.280260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.280268 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:38.280286 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.280287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.280296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.280355 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.280730 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.281185 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.282204 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.282466 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.282540 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.283559 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:38.283560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:38.283561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:38.283568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:38.285157 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.285502 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.285504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.285504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.285560 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.285747 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.285748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.285748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.285883 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:38.285955 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.286126 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.286209 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:55:38.305330 17356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:38.299525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.9.141.126:34219)
0925 14:55:38.299573 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.299575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.299586 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:38.299653 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.299654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.299672 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.299760 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.300342 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.300892 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.301681 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.301877 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.301975 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.305208 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34796: BlockingRecv error: recv got EOF from 127.0.0.1:34796 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:38.313707 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.306152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.9.141.126:34219)
0925 14:55:38.306212 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.306214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.306336 (+ 122us) server_negotiation.cc:407] Connection header received
0925 14:55:38.306417 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.306418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.306427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.306471 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.306940 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.307446 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.308112 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.308292 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.308350 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.310837 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:38.310839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:38.310839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:38.310845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:38.312330 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.312721 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.312721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.312722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.312771 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.313050 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.313051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.313052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.313167 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:38.313255 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.313484 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.313603 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:55:38.315608 17298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:38.317363 17298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:38.323935 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.319862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.9.141.67:39861)
0925 14:55:38.320258 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.320264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.320274 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:38.320315 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.320317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.320333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.320447 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.320623 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.321365 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.322802 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.323472 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.323571 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.323652 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.323798 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.323836 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":80,"threads_started":1}
I20260925 14:55:38.323928 17803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.319779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39861 (local address 127.0.0.1:37622)
0925 14:55:38.319962 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.320046 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.320117 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.320463 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.320465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.320468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.320537 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.320539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.321479 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.321480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.322715 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.322718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.323575 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.323575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.323666 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.323680 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.323780 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.323835 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":64,"threads_started":1}
I20260925 14:55:38.324183 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.319782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.0.0.1:52606)
0925 14:55:38.320091 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.320103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.320224 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.320533 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.320534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.320536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.320592 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.320594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.321866 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.321867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.322897 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.322899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.323931 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.323931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.323993 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.323997 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.324048 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.324100 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":79,"threads_started":1}
I20260925 14:55:38.324477 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.319895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.9.141.65:45437)
0925 14:55:38.320073 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.320079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.320192 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:55:38.320319 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.320321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.320337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.320469 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.320674 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.321776 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.322982 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.323748 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.323822 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.323903 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.324305 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.324347 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260925 14:55:38.325121 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 442b8a261e4a4d7984f28879578ebd01 (DEFAULT_TABLE table=test-table [id=a2c696e516904f0f89c25b159665ad61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:38.325152 17435 tablet_service.cc:1467] Processing CreateTablet for tablet 442b8a261e4a4d7984f28879578ebd01 (DEFAULT_TABLE table=test-table [id=a2c696e516904f0f89c25b159665ad61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:38.325492 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b8a261e4a4d7984f28879578ebd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.325501 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b8a261e4a4d7984f28879578ebd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.327989 17809 tablet_bootstrap.cc:492] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Bootstrap starting.
I20260925 14:55:38.328176 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.319988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:32775 (local address 127.0.0.1:50286)
0925 14:55:38.320198 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.320207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.320249 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.320768 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.320769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.320770 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.320801 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.320802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.322038 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.322039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.323025 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.323028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.323918 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.323919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.328015 (+ 4096us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.328024 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.328059 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.328106 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":71,"threads_started":1}
I20260925 14:55:38.328706 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.320215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.9.141.66:32775)
0925 14:55:38.320605 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.320610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.320620 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:38.320656 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.320658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.320673 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.320867 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.320921 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.321952 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.323108 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.323823 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.328372 (+ 4549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.328520 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.328552 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.328606 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":92,"threads_started":1}
I20260925 14:55:38.328912 17809 tablet_bootstrap.cc:654] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.328946 17810 tablet_bootstrap.cc:492] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Bootstrap starting.
I20260925 14:55:38.329223 17809 log.cc:824] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:38.329346 17575 tablet_service.cc:1467] Processing CreateTablet for tablet 442b8a261e4a4d7984f28879578ebd01 (DEFAULT_TABLE table=test-table [id=a2c696e516904f0f89c25b159665ad61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:38.329535 17810 tablet_bootstrap.cc:654] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.329694 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b8a261e4a4d7984f28879578ebd01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.329850 17810 log.cc:824] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:38.330046 17809 tablet_bootstrap.cc:492] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: No bootstrap required, opened a new log
I20260925 14:55:38.330122 17809 ts_tablet_manager.cc:1397] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:55:38.330581 17810 tablet_bootstrap.cc:492] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: No bootstrap required, opened a new log
I20260925 14:55:38.330701 17810 ts_tablet_manager.cc:1397] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:38.331593 17809 raft_consensus.cc:348] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.331754 17809 raft_consensus.cc:374] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.331795 17809 raft_consensus.cc:729] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.331921 17809 consensus_queue.cc:260] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.332201 17809 ts_tablet_manager.cc:1428] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:55:38.332358 17781 heartbeater.cc:503] Master 127.9.141.126:34219 was elected leader, sending a full tablet report...
I20260925 14:55:38.333043 17810 raft_consensus.cc:348] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.333209 17810 raft_consensus.cc:374] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.333247 17810 raft_consensus.cc:729] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.333364 17810 consensus_queue.cc:260] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.333640 17810 ts_tablet_manager.cc:1428] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:55:38.333663 17501 heartbeater.cc:503] Master 127.9.141.126:34219 was elected leader, sending a full tablet report...
I20260925 14:55:38.333990 17815 tablet_bootstrap.cc:492] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Bootstrap starting.
I20260925 14:55:38.334580 17815 tablet_bootstrap.cc:654] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.334895 17815 log.cc:824] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:38.335758 17815 tablet_bootstrap.cc:492] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: No bootstrap required, opened a new log
I20260925 14:55:38.335840 17815 ts_tablet_manager.cc:1397] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:55:38.337348 17815 raft_consensus.cc:348] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.337500 17815 raft_consensus.cc:374] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.337545 17815 raft_consensus.cc:729] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9e20d1ce642ef80ba45e3013a43b0, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.337653 17815 consensus_queue.cc:260] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.337898 17815 ts_tablet_manager.cc:1428] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:55:38.337970 17641 heartbeater.cc:503] Master 127.9.141.126:34219 was elected leader, sending a full tablet report...
W20260925 14:55:38.351841 17642 tablet.cc:2369] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:38.441466 17502 tablet.cc:2369] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:55:38.527432 17782 tablet.cc:2369] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:55:38.662137 17813 raft_consensus.cc:482] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:38.662250 17813 raft_consensus.cc:504] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.662740 17813 leader_election.cc:288] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437), 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775)
I20260925 14:55:38.666266 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.663220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:48483 (local address 127.9.141.65:45437)
0925 14:55:38.663269 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.663271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.663282 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:38.663314 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.663315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.663325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.663387 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.663623 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.664191 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.665488 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.665873 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.665933 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.665947 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.666141 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.666188 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:55:38.666292 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.663223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:35937 (local address 127.9.141.66:32775)
0925 14:55:38.663287 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.663289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.663323 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:55:38.663430 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.663431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.663441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.663566 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.663790 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.664339 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.665533 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.665937 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.666016 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.666024 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.666174 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.666215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:55:38.666325 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.663121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:32775 (local address 127.9.141.67:35937)
0925 14:55:38.663265 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.663275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.663337 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.663621 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.663622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.663624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.663705 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.663707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.664518 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.664519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.665410 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.665412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.666028 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.666029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.666080 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.666083 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.666156 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.666193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260925 14:55:38.666338 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.663007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.9.141.67:48483)
0925 14:55:38.663113 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.663128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.663201 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.663416 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.663428 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.663431 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.663512 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.663515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.664281 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.664282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.665404 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.665407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.665967 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.665968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.666069 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.666073 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.666119 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.666185 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":34}
I20260925 14:55:38.666673 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b8a261e4a4d7984f28879578ebd01" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" is_pre_election: true
I20260925 14:55:38.666862 17595 raft_consensus.cc:2454] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 0.
I20260925 14:55:38.666855 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b8a261e4a4d7984f28879578ebd01" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99acd84aace4b078e6f751d87d3ca18" is_pre_election: true
I20260925 14:55:38.667078 17455 raft_consensus.cc:2454] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 0.
I20260925 14:55:38.667145 17668 leader_election.cc:302] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, 9e8247c1bf4c48129e1bbb5ac1aa6e21; no voters:
I20260925 14:55:38.667289 17813 raft_consensus.cc:2781] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:38.667346 17813 raft_consensus.cc:482] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:38.667372 17813 raft_consensus.cc:3037] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.668366 17813 raft_consensus.cc:504] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.668566 17813 leader_election.cc:288] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 election: Requested vote from peers e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437), 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775)
I20260925 14:55:38.668706 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b8a261e4a4d7984f28879578ebd01" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99acd84aace4b078e6f751d87d3ca18"
I20260925 14:55:38.668749 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b8a261e4a4d7984f28879578ebd01" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
I20260925 14:55:38.668805 17455 raft_consensus.cc:3037] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.668843 17595 raft_consensus.cc:3037] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.669445 17455 raft_consensus.cc:2454] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 1.
I20260925 14:55:38.669510 17595 raft_consensus.cc:2454] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 1.
I20260925 14:55:38.669734 17668 leader_election.cc:302] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, 9e8247c1bf4c48129e1bbb5ac1aa6e21; no voters:
I20260925 14:55:38.669857 17813 raft_consensus.cc:2781] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:38.670037 17813 raft_consensus.cc:686] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Running, Role: LEADER
I20260925 14:55:38.670169 17813 consensus_queue.cc:237] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } }
I20260925 14:55:38.671026 17298 catalog_manager.cc:5495] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:38.676999 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.674365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.9.141.67:39861)
0925 14:55:38.674438 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.674440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.674448 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:38.674478 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.674479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.674498 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.674571 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:38.674701 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.675315 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.676190 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.676429 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.676497 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.676669 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:38.676756 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:38.676828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.676909 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.676927 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:55:38.678777 17455 raft_consensus.cc:1261] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Refusing update from remote peer 9e8247c1bf4c48129e1bbb5ac1aa6e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:38.678849 17595 raft_consensus.cc:1261] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Refusing update from remote peer 9e8247c1bf4c48129e1bbb5ac1aa6e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:38.679101 17813 consensus_queue.cc:1035] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:38.679235 17819 consensus_queue.cc:1035] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:38.681614 17823 mvcc.cc:205] Tried to move back new op lower bound from 7333265976024576000 to 7333265975992627200. Current Snapshot: MvccSnapshot[applied={T|T < 7333265976024576000}]
I20260925 14:55:38.683645 17298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:38.683743 17298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:38.684453 17298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Sending DeleteTablet for 3 replicas of tablet 442b8a261e4a4d7984f28879578ebd01
I20260925 14:55:38.684749 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b8a261e4a4d7984f28879578ebd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:38 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:52606
I20260925 14:55:38.684846 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b8a261e4a4d7984f28879578ebd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:38 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:37622
I20260925 14:55:38.685032 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b8a261e4a4d7984f28879578ebd01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:38 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:50286
I20260925 14:55:38.685029 17298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:38.685165 17298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:38.685207 17833 tablet_replica.cc:330] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: stopping tablet replica
I20260925 14:55:38.685400 17831 tablet_replica.cc:330] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: stopping tablet replica
I20260925 14:55:38.685453 17832 tablet_replica.cc:330] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: stopping tablet replica
I20260925 14:55:38.685540 17832 raft_consensus.cc:2229] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:38.685609 17831 raft_consensus.cc:2229] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:38.685695 17833 raft_consensus.cc:2229] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:38.685738 17831 raft_consensus.cc:2258] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:38.685811 17833 raft_consensus.cc:2258] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:38.686457 17833 ts_tablet_manager.cc:1905] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:38.686584 17831 ts_tablet_manager.cc:1905] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:38.686650 17435 tablet_service.cc:1467] Processing CreateTablet for tablet ba0e9a460e2d4b13b524c8b7f1337eb4 (DEFAULT_TABLE table=test-table [id=9f7f03aa0f6c451bae8d937edc0d85b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:38.686820 17575 tablet_service.cc:1467] Processing CreateTablet for tablet ba0e9a460e2d4b13b524c8b7f1337eb4 (DEFAULT_TABLE table=test-table [id=9f7f03aa0f6c451bae8d937edc0d85b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:38.686919 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0e9a460e2d4b13b524c8b7f1337eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.688197 17815 tablet_bootstrap.cc:492] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Bootstrap starting.
I20260925 14:55:38.688581 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0e9a460e2d4b13b524c8b7f1337eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.688578 17715 tablet_service.cc:1467] Processing CreateTablet for tablet ba0e9a460e2d4b13b524c8b7f1337eb4 (DEFAULT_TABLE table=test-table [id=9f7f03aa0f6c451bae8d937edc0d85b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:38.688634 17815 tablet_bootstrap.cc:654] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.688680 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba0e9a460e2d4b13b524c8b7f1337eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.689160 17815 tablet_bootstrap.cc:492] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: No bootstrap required, opened a new log
I20260925 14:55:38.689203 17815 ts_tablet_manager.cc:1397] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:38.689342 17815 raft_consensus.cc:348] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.689416 17815 raft_consensus.cc:374] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.689440 17815 raft_consensus.cc:729] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9e20d1ce642ef80ba45e3013a43b0, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.689494 17815 consensus_queue.cc:260] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.689590 17815 ts_tablet_manager.cc:1428] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:38.689924 17810 tablet_bootstrap.cc:492] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Bootstrap starting.
I20260925 14:55:38.689978 17809 tablet_bootstrap.cc:492] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Bootstrap starting.
I20260925 14:55:38.690263 17810 tablet_bootstrap.cc:654] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.690385 17809 tablet_bootstrap.cc:654] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.690950 17810 tablet_bootstrap.cc:492] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: No bootstrap required, opened a new log
I20260925 14:55:38.691016 17810 ts_tablet_manager.cc:1397] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:38.691188 17810 raft_consensus.cc:348] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.691262 17810 raft_consensus.cc:374] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.691284 17810 raft_consensus.cc:729] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.691332 17810 consensus_queue.cc:260] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.691431 17810 ts_tablet_manager.cc:1428] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:38.691618 17832 raft_consensus.cc:2258] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:38.692080 17833 ts_tablet_manager.cc:1918] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:38.692173 17833 log.cc:1196] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/442b8a261e4a4d7984f28879578ebd01
I20260925 14:55:38.692162 17832 ts_tablet_manager.cc:1905] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:38.692821 17833 ts_tablet_manager.cc:1939] T 442b8a261e4a4d7984f28879578ebd01 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting consensus metadata
I20260925 14:55:38.692813 17831 ts_tablet_manager.cc:1918] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:38.692946 17831 log.cc:1196] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/442b8a261e4a4d7984f28879578ebd01
I20260925 14:55:38.693250 17831 ts_tablet_manager.cc:1939] T 442b8a261e4a4d7984f28879578ebd01 P e99acd84aace4b078e6f751d87d3ca18: Deleting consensus metadata
I20260925 14:55:38.693222 17284 catalog_manager.cc:4841] TS 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775): tablet 442b8a261e4a4d7984f28879578ebd01 (table test-table [id=a2c696e516904f0f89c25b159665ad61]) successfully deleted
I20260925 14:55:38.693668 17832 ts_tablet_manager.cc:1918] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:38.693677 17284 catalog_manager.cc:4841] TS e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437): tablet 442b8a261e4a4d7984f28879578ebd01 (table test-table [id=a2c696e516904f0f89c25b159665ad61]) successfully deleted
I20260925 14:55:38.693750 17832 log.cc:1196] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/442b8a261e4a4d7984f28879578ebd01
I20260925 14:55:38.693992 17832 ts_tablet_manager.cc:1939] T 442b8a261e4a4d7984f28879578ebd01 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting consensus metadata
I20260925 14:55:38.696831 17809 tablet_bootstrap.cc:492] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: No bootstrap required, opened a new log
I20260925 14:55:38.696904 17809 ts_tablet_manager.cc:1397] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 14:55:38.697055 17809 raft_consensus.cc:348] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.697162 17809 raft_consensus.cc:374] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.697187 17809 raft_consensus.cc:729] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.697242 17809 consensus_queue.cc:260] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.697340 17809 ts_tablet_manager.cc:1428] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.699189 17282 catalog_manager.cc:4841] TS 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861): tablet 442b8a261e4a4d7984f28879578ebd01 (table test-table [id=a2c696e516904f0f89c25b159665ad61]) successfully deleted
I20260925 14:55:38.776789 17814 raft_consensus.cc:482] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:38.776885 17814 raft_consensus.cc:504] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.777249 17814 leader_election.cc:288] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775), 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:38.781237 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.777596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:57975 (local address 127.9.141.67:39861)
0925 14:55:38.777641 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.777644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.777759 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:55:38.777864 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.777866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.777876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.777908 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.778125 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.778824 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.780157 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.780813 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.780899 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.780908 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.781116 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.781158 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:55:38.781284 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.777454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:32775 (local address 127.9.141.65:53509)
0925 14:55:38.777614 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.777628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.777690 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.777941 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.777942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.777944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.778063 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.778065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.779022 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.779023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.780151 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.780154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.780946 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.780946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.781032 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.781036 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.781123 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.781162 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260925 14:55:38.781327 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.777609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:53509 (local address 127.9.141.66:32775)
0925 14:55:38.777655 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.777657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.777673 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:55:38.777827 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.777828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.777839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.777880 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:38.778199 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.778912 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.780228 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.780836 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.780900 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.780908 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.781216 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.781261 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:38.781481 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.777456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39861 (local address 127.9.141.65:57975)
0925 14:55:38.777699 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.777708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.777784 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.777975 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.777976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.777977 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:38.778039 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.778041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.778916 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.778918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.780059 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.780061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.780937 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.780939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.781022 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.781027 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.781094 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.781136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":38,"thread_start_us":93,"threads_started":1}
I20260925 14:55:38.781773 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0e9a460e2d4b13b524c8b7f1337eb4" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" is_pre_election: true
I20260925 14:55:38.781873 17595 raft_consensus.cc:2454] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 0.
I20260925 14:55:38.781930 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0e9a460e2d4b13b524c8b7f1337eb4" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" is_pre_election: true
I20260925 14:55:38.782066 17735 raft_consensus.cc:2454] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 0.
I20260925 14:55:38.782249 17388 leader_election.cc:302] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:38.782395 17814 raft_consensus.cc:2781] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:38.782444 17814 raft_consensus.cc:482] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:38.782466 17814 raft_consensus.cc:3037] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.783201 17814 raft_consensus.cc:504] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.783357 17814 leader_election.cc:288] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 election: Requested vote from peers 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775), 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:38.783514 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0e9a460e2d4b13b524c8b7f1337eb4" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21"
I20260925 14:55:38.783517 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba0e9a460e2d4b13b524c8b7f1337eb4" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
I20260925 14:55:38.783593 17735 raft_consensus.cc:3037] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.783613 17595 raft_consensus.cc:3037] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.784260 17595 raft_consensus.cc:2454] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 1.
I20260925 14:55:38.784477 17735 raft_consensus.cc:2454] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 1.
I20260925 14:55:38.784499 17388 leader_election.cc:302] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:38.784601 17814 raft_consensus.cc:2781] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:38.784752 17814 raft_consensus.cc:686] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 1 LEADER]: Becoming Leader. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Running, Role: LEADER
I20260925 14:55:38.784827 17814 consensus_queue.cc:237] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.785670 17298 catalog_manager.cc:5495] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 reported cstate change: term changed from 0 to 1, leader changed from <none> to e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65). New cstate: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348137, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.126@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:38.828673 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.821086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:34219 (local address 127.0.0.1:34818)
0925 14:55:38.821254 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.821270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.821356 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.821590 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.821591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.821919 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:38.821977 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.821979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.822686 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.822687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.823414 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.823416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.823800 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.823801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.823845 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.824565 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:38.824570 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:38.825916 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:38.827554 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.827560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.827563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.827781 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.827973 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:38.827974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:38.827974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:38.828030 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:38.828310 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:38.828315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:38.828357 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.828474 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.828598 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260925 14:55:38.828811 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.821151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.9.141.126:34219)
0925 14:55:38.821340 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.821343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.821353 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:38.821413 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.821413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.821425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.821517 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:38.822068 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.822572 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.823522 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.823720 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.823781 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.826021 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:38.826022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:38.826022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:38.826029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:38.827467 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.827854 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.827854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.827855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.827909 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:38.828119 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:38.828120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:38.828121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:38.828238 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:38.828305 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.828543 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.828668 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260925 14:55:38.831046 17298 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:34818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:55:38.831236 17298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:38.832999 17575 tablet_service.cc:1467] Processing CreateTablet for tablet ac76e1453863426ab1b27a668f9b77c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cceed41a13e4e5aba3bf9fd9347c15f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:38.832993 17715 tablet_service.cc:1467] Processing CreateTablet for tablet ac76e1453863426ab1b27a668f9b77c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cceed41a13e4e5aba3bf9fd9347c15f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:38.833024 17435 tablet_service.cc:1467] Processing CreateTablet for tablet ac76e1453863426ab1b27a668f9b77c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7cceed41a13e4e5aba3bf9fd9347c15f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:55:38.833158 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac76e1453863426ab1b27a668f9b77c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.833156 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac76e1453863426ab1b27a668f9b77c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.833197 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac76e1453863426ab1b27a668f9b77c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:38.834708 17815 tablet_bootstrap.cc:492] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: Bootstrap starting.
I20260925 14:55:38.835127 17815 tablet_bootstrap.cc:654] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.835431 17809 tablet_bootstrap.cc:492] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Bootstrap starting.
I20260925 14:55:38.835708 17809 tablet_bootstrap.cc:654] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.835783 17815 tablet_bootstrap.cc:492] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: No bootstrap required, opened a new log
I20260925 14:55:38.835821 17815 ts_tablet_manager.cc:1397] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:38.835917 17810 tablet_bootstrap.cc:492] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: Bootstrap starting.
I20260925 14:55:38.836004 17815 raft_consensus.cc:348] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.836090 17815 raft_consensus.cc:374] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.836117 17815 raft_consensus.cc:729] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9e20d1ce642ef80ba45e3013a43b0, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.836164 17815 consensus_queue.cc:260] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.836225 17809 tablet_bootstrap.cc:492] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: No bootstrap required, opened a new log
I20260925 14:55:38.836264 17809 ts_tablet_manager.cc:1397] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:38.836268 17815 ts_tablet_manager.cc:1428] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:55:38.836436 17810 tablet_bootstrap.cc:654] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:38.836408 17809 raft_consensus.cc:348] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.836477 17809 raft_consensus.cc:374] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.836498 17809 raft_consensus.cc:729] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.836462 17817 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.836535 17817 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.836537 17809 consensus_queue.cc:260] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.836619 17809 ts_tablet_manager.cc:1428] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.836730 17813 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.836791 17813 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.837218 17810 tablet_bootstrap.cc:492] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: No bootstrap required, opened a new log
I20260925 14:55:38.837275 17810 ts_tablet_manager.cc:1397] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:38.837395 17810 raft_consensus.cc:348] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.837447 17810 raft_consensus.cc:374] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:38.837460 17810 raft_consensus.cc:729] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Initialized, Role: FOLLOWER
I20260925 14:55:38.837530 17810 consensus_queue.cc:260] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.837651 17810 ts_tablet_manager.cc:1428] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:38.837695 17814 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.837785 17814 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.936959 17814 raft_consensus.cc:482] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:38.937107 17814 raft_consensus.cc:504] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.937282 17814 leader_election.cc:288] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775), 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:38.937556 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac76e1453863426ab1b27a668f9b77c3" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" is_pre_election: true
I20260925 14:55:38.937642 17595 raft_consensus.cc:2454] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 0.
I20260925 14:55:38.937559 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac76e1453863426ab1b27a668f9b77c3" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" is_pre_election: true
I20260925 14:55:38.937686 17735 raft_consensus.cc:2454] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 0.
I20260925 14:55:38.937827 17388 leader_election.cc:302] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:38.937935 17814 raft_consensus.cc:2781] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:38.937974 17814 raft_consensus.cc:482] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:38.937995 17814 raft_consensus.cc:3037] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.938684 17814 raft_consensus.cc:504] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.938836 17814 leader_election.cc:288] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 election: Requested vote from peers 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775), 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:38.939006 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac76e1453863426ab1b27a668f9b77c3" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
I20260925 14:55:38.939037 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac76e1453863426ab1b27a668f9b77c3" candidate_uuid: "e99acd84aace4b078e6f751d87d3ca18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21"
I20260925 14:55:38.939088 17595 raft_consensus.cc:3037] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.939100 17735 raft_consensus.cc:3037] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:38.939816 17735 raft_consensus.cc:2454] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 1.
I20260925 14:55:38.939816 17595 raft_consensus.cc:2454] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99acd84aace4b078e6f751d87d3ca18 in term 1.
I20260925 14:55:38.940050 17388 leader_election.cc:302] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:38.940243 17814 raft_consensus.cc:2781] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:38.940322 17814 raft_consensus.cc:686] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [term 1 LEADER]: Becoming Leader. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Running, Role: LEADER
I20260925 14:55:38.940410 17814 consensus_queue.cc:237] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:38.940814 17839 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: TxnStatusTablet state changed. Reason: New leader e99acd84aace4b078e6f751d87d3ca18. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.940932 17839 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:38.941114 17852 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:55:38.941141 17297 catalog_manager.cc:5495] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 reported cstate change: term changed from 0 to 1, leader changed from <none> to e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65). New cstate: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:38.941241 17852 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:55:38.954082 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.951213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.9.141.65:45437)
0925 14:55:38.951281 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.951283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.951296 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:55:38.951398 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.951399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.951418 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.951522 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:38.951656 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.952194 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.953152 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.953401 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.953497 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.953708 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:38.953822 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:38.953914 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.953976 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.953999 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:55:38.954113 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.951146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.0.0.1:52614)
0925 14:55:38.951252 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.951267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.951319 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.951526 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.951527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.951528 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:38.951585 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.951588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.952284 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.952286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.953027 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.953029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.953499 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.953500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.953564 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.953583 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:38.953897 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:38.953907 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.953969 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.954024 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 14:55:38.955238 17595 raft_consensus.cc:1261] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Refusing update from remote peer e99acd84aace4b078e6f751d87d3ca18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:38.955271 17735 raft_consensus.cc:1261] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Refusing update from remote peer e99acd84aace4b078e6f751d87d3ca18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:38.955550 17814 consensus_queue.cc:1035] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:38.955737 17814 consensus_queue.cc:1035] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:38.956351 17817 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: TxnStatusTablet state changed. Reason: New leader e99acd84aace4b078e6f751d87d3ca18. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.956471 17817 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.956430 17813 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: TxnStatusTablet state changed. Reason: New leader e99acd84aace4b078e6f751d87d3ca18. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.956524 17813 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.957267 17814 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.957370 17814 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:38.957434 17824 mvcc.cc:205] Tried to move back new op lower bound from 7333265977158516736 to 7333265977099120640. Current Snapshot: MvccSnapshot[applied={T|T < 7333265977158516736 or (T in {7333265977158516736})}]
I20260925 14:55:38.957967 17814 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.958055 17814 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P e99acd84aace4b078e6f751d87d3ca18: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:55:38.958143 17813 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.958226 17813 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.958359 17822 mvcc.cc:205] Tried to move back new op lower bound from 7333265977158516736 to 7333265977099120640. Current Snapshot: MvccSnapshot[applied={T|T < 7333265977158516736 or (T in {7333265977158516736})}]
I20260925 14:55:38.959002 17817 tablet_replica.cc:439] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e99acd84aace4b078e6f751d87d3ca18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } } }
I20260925 14:55:38.959095 17817 tablet_replica.cc:442] T ac76e1453863426ab1b27a668f9b77c3 P 7ed9e20d1ce642ef80ba45e3013a43b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:55:38.962035 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.959832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.9.141.65:45437)
0925 14:55:38.959866 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.959868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.959875 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:38.959906 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.959907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.959915 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.959959 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:38.960067 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.960613 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.961335 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.961545 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.961622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.961776 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:38.961808 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:38.961876 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.961940 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.961958 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:55:38.967612 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.964507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:35885 (local address 127.9.141.65:45437)
0925 14:55:38.964551 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:38.964553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:38.964691 (+ 138us) server_negotiation.cc:407] Connection header received
0925 14:55:38.964736 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:38.964737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:38.964747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:38.964810 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:38.964933 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.965713 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.966953 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.967126 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.967176 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.967371 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:38.967394 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:38.967434 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:38.967522 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 14:55:38.967538 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:38.967762 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:38.964443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.9.141.65:35885)
0925 14:55:38.964622 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:38.964630 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:38.964681 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:38.964813 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:38.964814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:38.964822 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:38.964870 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.964871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.965809 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.965810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.966857 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:38.966859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:38.967198 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:38.967199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:38.967273 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:38.967288 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:38.967456 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:38.967458 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:38.967510 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:55:38.967558 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260925 14:55:38.970014 17735 raft_consensus.cc:1261] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Refusing update from remote peer e99acd84aace4b078e6f751d87d3ca18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:38.970187 17595 raft_consensus.cc:1261] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Refusing update from remote peer e99acd84aace4b078e6f751d87d3ca18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:38.970314 17814 consensus_queue.cc:1035] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:38.970409 17839 consensus_queue.cc:1035] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:38.982118 17296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:38.982234 17296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:38.982832 17296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Sending DeleteTablet for 3 replicas of tablet ba0e9a460e2d4b13b524c8b7f1337eb4
I20260925 14:55:38.983093 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0e9a460e2d4b13b524c8b7f1337eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:38 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:52606
I20260925 14:55:38.983165 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0e9a460e2d4b13b524c8b7f1337eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:38 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:50286
I20260925 14:55:38.983207 17831 tablet_replica.cc:330] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: stopping tablet replica
I20260925 14:55:38.983263 17831 raft_consensus.cc:2229] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:38.983333 17833 tablet_replica.cc:330] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: stopping tablet replica
I20260925 14:55:38.983366 17831 raft_consensus.cc:2258] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:38.983387 17833 raft_consensus.cc:2229] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:38.983433 17833 raft_consensus.cc:2258] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:38.983531 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet ba0e9a460e2d4b13b524c8b7f1337eb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:38 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:37622
I20260925 14:55:38.983640 17832 tablet_replica.cc:330] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: stopping tablet replica
I20260925 14:55:38.983696 17832 raft_consensus.cc:2229] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:38.983736 17832 raft_consensus.cc:2258] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:38.983830 17833 ts_tablet_manager.cc:1905] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:38.983879 17831 ts_tablet_manager.cc:1905] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:38.986788 17833 ts_tablet_manager.cc:1918] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:38.986860 17833 log.cc:1196] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/ba0e9a460e2d4b13b524c8b7f1337eb4
I20260925 14:55:38.987105 17833 ts_tablet_manager.cc:1939] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting consensus metadata
I20260925 14:55:38.987277 17831 ts_tablet_manager.cc:1918] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:38.987337 17831 log.cc:1196] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/ba0e9a460e2d4b13b524c8b7f1337eb4
I20260925 14:55:38.987468 17284 catalog_manager.cc:4841] TS 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775): tablet ba0e9a460e2d4b13b524c8b7f1337eb4 (table test-table [id=9f7f03aa0f6c451bae8d937edc0d85b7]) successfully deleted
I20260925 14:55:38.987586 17831 ts_tablet_manager.cc:1939] T ba0e9a460e2d4b13b524c8b7f1337eb4 P e99acd84aace4b078e6f751d87d3ca18: Deleting consensus metadata
I20260925 14:55:38.987581 17832 ts_tablet_manager.cc:1905] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:38.988006 17284 catalog_manager.cc:4841] TS e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437): tablet ba0e9a460e2d4b13b524c8b7f1337eb4 (table test-table [id=9f7f03aa0f6c451bae8d937edc0d85b7]) successfully deleted
I20260925 14:55:38.989099 17832 ts_tablet_manager.cc:1918] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:38.989176 17832 log.cc:1196] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/ba0e9a460e2d4b13b524c8b7f1337eb4
I20260925 14:55:38.989410 17832 ts_tablet_manager.cc:1939] T ba0e9a460e2d4b13b524c8b7f1337eb4 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting consensus metadata
I20260925 14:55:38.989957 17282 catalog_manager.cc:4841] TS 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861): tablet ba0e9a460e2d4b13b524c8b7f1337eb4 (table test-table [id=9f7f03aa0f6c451bae8d937edc0d85b7]) successfully deleted
Sep 25 14:55:38 dist-test-slave-1sv7 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:55:39 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:39 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:55:39.010859 17847 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:39.003731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.9.141.126:34219)
0925 14:55:39.003802 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.003805 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.003974 (+ 169us) server_negotiation.cc:407] Connection header received
0925 14:55:39.004129 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.004130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.004145 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.004223 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:39.005020 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.006140 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.007226 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.007593 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.007676 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.010783 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34832: BlockingRecv error: recv got EOF from 127.0.0.1:34832 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 14:55:39 dist-test-slave-1sv7 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM
I20260925 14:55:39.019970 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.011674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.9.141.126:34219)
0925 14:55:39.011713 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.011715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.011953 (+ 238us) server_negotiation.cc:407] Connection header received
0925 14:55:39.011983 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.011984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.011997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.012062 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:39.012729 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.013512 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.014304 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.014640 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.014723 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.017019 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:39.017021 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:39.017022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:39.017029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:39.018715 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:39.019124 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:39.019125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:39.019125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:39.019197 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:39.019432 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:39.019433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:39.019434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:39.019574 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:39.019683 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.019772 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.019863 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:39.021634 17296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:55:39.021838 17296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:39.023290 17435 tablet_service.cc:1467] Processing CreateTablet for tablet 32ca68ebff8240b486a971f007863e65 (DEFAULT_TABLE table=test-table [id=a06abe8a69ce4414adac29badd5c8fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:39.023357 17575 tablet_service.cc:1467] Processing CreateTablet for tablet 32ca68ebff8240b486a971f007863e65 (DEFAULT_TABLE table=test-table [id=a06abe8a69ce4414adac29badd5c8fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:39.023453 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ca68ebff8240b486a971f007863e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:39.023492 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ca68ebff8240b486a971f007863e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:39.025025 17810 tablet_bootstrap.cc:492] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Bootstrap starting.
I20260925 14:55:39.025024 17815 tablet_bootstrap.cc:492] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Bootstrap starting.
I20260925 14:55:39.025089 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 32ca68ebff8240b486a971f007863e65 (DEFAULT_TABLE table=test-table [id=a06abe8a69ce4414adac29badd5c8fcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:39.025206 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ca68ebff8240b486a971f007863e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:39.025450 17810 tablet_bootstrap.cc:654] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:39.025480 17815 tablet_bootstrap.cc:654] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:39.026376 17810 tablet_bootstrap.cc:492] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: No bootstrap required, opened a new log
I20260925 14:55:39.026422 17810 ts_tablet_manager.cc:1397] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:39.026465 17815 tablet_bootstrap.cc:492] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: No bootstrap required, opened a new log
I20260925 14:55:39.026510 17815 ts_tablet_manager.cc:1397] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:39.026569 17810 raft_consensus.cc:348] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.026643 17810 raft_consensus.cc:374] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:39.026662 17810 raft_consensus.cc:729] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Initialized, Role: FOLLOWER
I20260925 14:55:39.026656 17815 raft_consensus.cc:348] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.026728 17815 raft_consensus.cc:374] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:39.026710 17810 consensus_queue.cc:260] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.026751 17815 raft_consensus.cc:729] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9e20d1ce642ef80ba45e3013a43b0, State: Initialized, Role: FOLLOWER
I20260925 14:55:39.026818 17810 ts_tablet_manager.cc:1428] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:39.026796 17815 consensus_queue.cc:260] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.026894 17815 ts_tablet_manager.cc:1428] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:39.026986 17809 tablet_bootstrap.cc:492] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Bootstrap starting.
I20260925 14:55:39.027303 17809 tablet_bootstrap.cc:654] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:39.028018 17809 tablet_bootstrap.cc:492] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: No bootstrap required, opened a new log
I20260925 14:55:39.028110 17809 ts_tablet_manager.cc:1397] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:39.028239 17809 raft_consensus.cc:348] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.028316 17809 raft_consensus.cc:374] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:39.028332 17809 raft_consensus.cc:729] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Initialized, Role: FOLLOWER
I20260925 14:55:39.028575 17809 consensus_queue.cc:260] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.028721 17809 ts_tablet_manager.cc:1428] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:39.031670 17813 raft_consensus.cc:482] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:39.031739 17813 raft_consensus.cc:504] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.031844 17813 leader_election.cc:288] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437), 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775)
I20260925 14:55:39.032086 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ca68ebff8240b486a971f007863e65" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99acd84aace4b078e6f751d87d3ca18" is_pre_election: true
I20260925 14:55:39.032204 17455 raft_consensus.cc:2454] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 0.
I20260925 14:55:39.032318 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ca68ebff8240b486a971f007863e65" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" is_pre_election: true
I20260925 14:55:39.032363 17668 leader_election.cc:302] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8247c1bf4c48129e1bbb5ac1aa6e21, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:39.032449 17595 raft_consensus.cc:2454] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 0.
I20260925 14:55:39.032497 17813 raft_consensus.cc:2781] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:39.032526 17813 raft_consensus.cc:482] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:39.032552 17813 raft_consensus.cc:3037] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:39.033317 17813 raft_consensus.cc:504] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.033434 17813 leader_election.cc:288] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 election: Requested vote from peers e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437), 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775)
I20260925 14:55:39.033633 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ca68ebff8240b486a971f007863e65" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99acd84aace4b078e6f751d87d3ca18"
I20260925 14:55:39.033682 17595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ca68ebff8240b486a971f007863e65" candidate_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0"
I20260925 14:55:39.033728 17455 raft_consensus.cc:3037] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:39.033746 17595 raft_consensus.cc:3037] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:39.034375 17455 raft_consensus.cc:2454] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 1.
I20260925 14:55:39.034525 17595 raft_consensus.cc:2454] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8247c1bf4c48129e1bbb5ac1aa6e21 in term 1.
I20260925 14:55:39.034554 17668 leader_election.cc:302] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e8247c1bf4c48129e1bbb5ac1aa6e21, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:39.034715 17813 raft_consensus.cc:2781] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:39.034794 17813 raft_consensus.cc:686] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Running, Role: LEADER
I20260925 14:55:39.034847 17813 consensus_queue.cc:237] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.035377 17296 catalog_manager.cc:5495] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } health_report { overall_health: HEALTHY } } }
I20260925 14:55:39.041178 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.038467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.9.141.67:39861)
0925 14:55:39.038508 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.038510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.038519 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:39.038553 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.038554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.038561 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.038612 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:39.038728 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.039196 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.040355 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.040610 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.040674 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.040890 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:39.040918 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:39.040969 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.041101 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.041124 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:55:39.041990 17595 raft_consensus.cc:1261] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Refusing update from remote peer 9e8247c1bf4c48129e1bbb5ac1aa6e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:39.042016 17455 raft_consensus.cc:1261] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Refusing update from remote peer 9e8247c1bf4c48129e1bbb5ac1aa6e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:39.042243 17813 consensus_queue.cc:1035] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:39.042358 17819 consensus_queue.cc:1035] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:39.044932 17297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:39.045023 17297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:39.045751 17297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Sending DeleteTablet for 3 replicas of tablet 32ca68ebff8240b486a971f007863e65
I20260925 14:55:39.045982 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ca68ebff8240b486a971f007863e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:52606
I20260925 14:55:39.046074 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ca68ebff8240b486a971f007863e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:37622
I20260925 14:55:39.046164 17831 tablet_replica.cc:330] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: stopping tablet replica
I20260925 14:55:39.046129 17575 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ca68ebff8240b486a971f007863e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:50286
I20260925 14:55:39.046144 17574 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ca68ebff8240b486a971f007863e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:50286
I20260925 14:55:39.046223 17831 raft_consensus.cc:2229] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:39.046241 17833 tablet_replica.cc:330] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: stopping tablet replica
I20260925 14:55:39.046267 17831 raft_consensus.cc:2258] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:39.046293 17833 raft_consensus.cc:2229] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:39.046334 17833 raft_consensus.cc:2258] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:39.046340 17832 tablet_replica.cc:330] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: stopping tablet replica
I20260925 14:55:39.046396 17832 raft_consensus.cc:2229] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:39.046456 17832 raft_consensus.cc:2258] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:39.046610 17833 ts_tablet_manager.cc:1905] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:39.046813 17831 ts_tablet_manager.cc:1905] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:39.047993 17832 ts_tablet_manager.cc:1905] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:39.048236 17831 ts_tablet_manager.cc:1918] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:39.048290 17831 log.cc:1196] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/32ca68ebff8240b486a971f007863e65
I20260925 14:55:39.048553 17831 ts_tablet_manager.cc:1939] T 32ca68ebff8240b486a971f007863e65 P e99acd84aace4b078e6f751d87d3ca18: Deleting consensus metadata
I20260925 14:55:39.048720 17833 ts_tablet_manager.cc:1918] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:55:39.048766 17833 log.cc:1196] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/32ca68ebff8240b486a971f007863e65
I20260925 14:55:39.048977 17833 ts_tablet_manager.cc:1939] T 32ca68ebff8240b486a971f007863e65 P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting consensus metadata
I20260925 14:55:39.049404 17297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 14:55:39.049499 17832 ts_tablet_manager.cc:1918] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 14:55:39.049556 17297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:55:39.049556 17832 log.cc:1196] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/32ca68ebff8240b486a971f007863e65
I20260925 14:55:39.049782 17832 ts_tablet_manager.cc:1939] T 32ca68ebff8240b486a971f007863e65 P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting consensus metadata
I20260925 14:55:39.050149 17284 catalog_manager.cc:4841] TS e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437): tablet 32ca68ebff8240b486a971f007863e65 (table test-table [id=a06abe8a69ce4414adac29badd5c8fcd]) successfully deleted
I20260925 14:55:39.050551 17284 catalog_manager.cc:4841] TS 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775): tablet 32ca68ebff8240b486a971f007863e65 (table test-table [id=a06abe8a69ce4414adac29badd5c8fcd]) successfully deleted
W20260925 14:55:39.050715 17284 catalog_manager.cc:4808] TS 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775): delete failed for tablet 32ca68ebff8240b486a971f007863e65 because the tablet was not found. No further retry: Not found: Tablet not found: 32ca68ebff8240b486a971f007863e65
I20260925 14:55:39.050869 17715 tablet_service.cc:1467] Processing CreateTablet for tablet 1227f54d6ca8499d860df9d4103abf6d (DEFAULT_TABLE table=test-table [id=9e0adc77ebe446f485f6b2bcdfdc0c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:39.050979 17715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1227f54d6ca8499d860df9d4103abf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:39.051009 17574 tablet_service.cc:1467] Processing CreateTablet for tablet 1227f54d6ca8499d860df9d4103abf6d (DEFAULT_TABLE table=test-table [id=9e0adc77ebe446f485f6b2bcdfdc0c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:39.051110 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1227f54d6ca8499d860df9d4103abf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:39.052294 17809 tablet_bootstrap.cc:492] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Bootstrap starting.
I20260925 14:55:39.052492 17435 tablet_service.cc:1467] Processing CreateTablet for tablet 1227f54d6ca8499d860df9d4103abf6d (DEFAULT_TABLE table=test-table [id=9e0adc77ebe446f485f6b2bcdfdc0c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:55:39.052603 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1227f54d6ca8499d860df9d4103abf6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:39.052651 17809 tablet_bootstrap.cc:654] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:39.052886 17815 tablet_bootstrap.cc:492] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Bootstrap starting.
I20260925 14:55:39.053217 17815 tablet_bootstrap.cc:654] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:39.053831 17815 tablet_bootstrap.cc:492] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: No bootstrap required, opened a new log
I20260925 14:55:39.053879 17815 ts_tablet_manager.cc:1397] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:39.053925 17810 tablet_bootstrap.cc:492] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: Bootstrap starting.
I20260925 14:55:39.054039 17282 catalog_manager.cc:4841] TS 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861): tablet 32ca68ebff8240b486a971f007863e65 (table test-table [id=a06abe8a69ce4414adac29badd5c8fcd]) successfully deleted
I20260925 14:55:39.054028 17815 raft_consensus.cc:348] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.054111 17815 raft_consensus.cc:374] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:39.054137 17815 raft_consensus.cc:729] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed9e20d1ce642ef80ba45e3013a43b0, State: Initialized, Role: FOLLOWER
I20260925 14:55:39.054188 17815 consensus_queue.cc:260] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.054291 17815 ts_tablet_manager.cc:1428] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:39.054381 17810 tablet_bootstrap.cc:654] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:39.055013 17809 tablet_bootstrap.cc:492] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: No bootstrap required, opened a new log
I20260925 14:55:39.055038 17810 tablet_bootstrap.cc:492] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: No bootstrap required, opened a new log
I20260925 14:55:39.055073 17809 ts_tablet_manager.cc:1397] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:55:39.055078 17810 ts_tablet_manager.cc:1397] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:39.055191 17809 raft_consensus.cc:348] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.055220 17810 raft_consensus.cc:348] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.055271 17809 raft_consensus.cc:374] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:39.055296 17809 raft_consensus.cc:729] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8247c1bf4c48129e1bbb5ac1aa6e21, State: Initialized, Role: FOLLOWER
I20260925 14:55:39.055296 17810 raft_consensus.cc:374] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:39.055320 17810 raft_consensus.cc:729] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99acd84aace4b078e6f751d87d3ca18, State: Initialized, Role: FOLLOWER
I20260925 14:55:39.055346 17809 consensus_queue.cc:260] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.055373 17810 consensus_queue.cc:260] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.055449 17809 ts_tablet_manager.cc:1428] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:39.055492 17810 ts_tablet_manager.cc:1428] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:39.071693 17817 raft_consensus.cc:482] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:55:39.071779 17817 raft_consensus.cc:504] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.072113 17817 leader_election.cc:288] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437), 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:39.075857 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.072306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:48119 (local address 127.9.141.65:45437)
0925 14:55:39.072351 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.072354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.072480 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:55:39.072607 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.072608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.072618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.072664 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:39.072836 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.073647 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.075073 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.075455 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.075551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.075563 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.075758 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.075789 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:55:39.075886 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.072264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.9.141.66:48119)
0925 14:55:39.072418 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.072429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:39.072520 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:39.072677 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:39.072678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:39.072680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:39.072752 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.072754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.073746 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.073747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.074989 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.074991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.075570 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.075571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.075669 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.075677 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:55:39.075749 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:55:39.075787 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260925 14:55:39.076129 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1227f54d6ca8499d860df9d4103abf6d" candidate_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99acd84aace4b078e6f751d87d3ca18" is_pre_election: true
I20260925 14:55:39.076233 17455 raft_consensus.cc:2454] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed9e20d1ce642ef80ba45e3013a43b0 in term 0.
I20260925 14:55:39.076424 17528 leader_election.cc:302] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:39.076570 17817 raft_consensus.cc:2781] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:55:39.076614 17817 raft_consensus.cc:482] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:55:39.076642 17817 raft_consensus.cc:3037] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:39.077292 17817 raft_consensus.cc:504] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.077431 17817 leader_election.cc:288] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [CANDIDATE]: Term 1 election: Requested vote from peers e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437), 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861)
I20260925 14:55:39.077641 17455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1227f54d6ca8499d860df9d4103abf6d" candidate_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99acd84aace4b078e6f751d87d3ca18"
I20260925 14:55:39.077728 17455 raft_consensus.cc:3037] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:39.077739 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.072303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39861 (local address 127.9.141.66:34477)
0925 14:55:39.072543 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.072555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:39.072645 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:39.073770 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:39.073771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:39.073773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:55:39.073848 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.073850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.074837 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.074838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.076564 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.076567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.077458 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.077459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.077539 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.077543 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:55:39.077609 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:55:39.077648 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":124,"threads_started":1}
I20260925 14:55:39.078361 17455 raft_consensus.cc:2454] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed9e20d1ce642ef80ba45e3013a43b0 in term 1.
I20260925 14:55:39.078593 17528 leader_election.cc:302] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ed9e20d1ce642ef80ba45e3013a43b0, e99acd84aace4b078e6f751d87d3ca18; no voters:
I20260925 14:55:39.078708 17817 raft_consensus.cc:2781] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:39.078734 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.072479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:34477 (local address 127.9.141.67:39861)
0925 14:55:39.072522 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.072524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.072615 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:55:39.072742 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.072743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.072756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.072812 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:55:39.073934 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.074704 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.076691 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.077305 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.077393 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.077403 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.078609 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.078660 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:55:39.078852 17817 raft_consensus.cc:686] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 LEADER]: Becoming Leader. State: Replica: 7ed9e20d1ce642ef80ba45e3013a43b0, State: Running, Role: LEADER
I20260925 14:55:39.078930 17817 consensus_queue.cc:237] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } }
I20260925 14:55:39.078962 17735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1227f54d6ca8499d860df9d4103abf6d" candidate_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" is_pre_election: true
I20260925 14:55:39.079087 17735 raft_consensus.cc:2454] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ed9e20d1ce642ef80ba45e3013a43b0 in term 0.
I20260925 14:55:39.078979 17734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1227f54d6ca8499d860df9d4103abf6d" candidate_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21"
I20260925 14:55:39.079242 17734 raft_consensus.cc:3037] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:39.079638 17298 catalog_manager.cc:5495] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ed9e20d1ce642ef80ba45e3013a43b0" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 32775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 } health_report { overall_health: UNKNOWN } } }
I20260925 14:55:39.080106 17734 raft_consensus.cc:2454] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ed9e20d1ce642ef80ba45e3013a43b0 in term 1.
I20260925 14:55:39.086360 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.084007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.9.141.66:32775)
0925 14:55:39.084071 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.084073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.084080 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:39.084109 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.084110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.084128 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.084216 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:39.084328 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.084838 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.085566 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.085777 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.085855 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.086020 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:39.086126 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:39.086215 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.086252 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.086272 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:55:39.092031 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.089059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.9.141.66:34905)
0925 14:55:39.089216 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.089228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:39.089294 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:39.089456 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:39.089457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:39.089458 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:39.089515 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.089516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.090490 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.090491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.091072 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.091074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.091621 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.091622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.091677 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.091691 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:39.091893 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:39.091895 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:39.091931 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 14:55:39.091966 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260925 14:55:39.092037 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.089200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:34905 (local address 127.9.141.65:45437)
0925 14:55:39.089271 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.089273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.089285 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:39.089363 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.089364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.089377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.089455 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:39.089576 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.090387 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.091168 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.091517 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.091599 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.091786 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:39.091825 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:39.091885 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.091934 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.091954 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:55:39.094664 17734 raft_consensus.cc:1261] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Refusing update from remote peer 7ed9e20d1ce642ef80ba45e3013a43b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:39.094723 17455 raft_consensus.cc:1261] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Refusing update from remote peer 7ed9e20d1ce642ef80ba45e3013a43b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:55:39.095021 17817 consensus_queue.cc:1035] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99acd84aace4b078e6f751d87d3ca18" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:39.095163 17888 consensus_queue.cc:1035] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e8247c1bf4c48129e1bbb5ac1aa6e21" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:55:39.103128 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.099912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:32775 (local address 127.9.141.65:34723)
0925 14:55:39.099958 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.099969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:39.100007 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:39.100142 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:39.100143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:39.100144 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:55:39.100188 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.100189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.101070 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.101071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.101996 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:39.101999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.102529 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.102530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:39.102611 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.102618 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:55:39.102949 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:55:39.102951 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:39.102996 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:55:39.103046 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:55:39.103413 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.099949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:34723 (local address 127.9.141.66:32775)
0925 14:55:39.099988 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:39.099991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:39.099999 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:39.100069 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:39.100070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:39.100079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:39.100134 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:55:39.100254 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.100977 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.102098 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:39.102435 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:39.102518 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:39.102831 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:55:39.102867 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:55:39.102922 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:39.103322 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 14:55:39.103347 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:55:39.113442 17298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:39.113550 17298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:55:39.114295 17298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7fe0fdef3154ca2bb4a4cdeefc9368a: Sending DeleteTablet for 3 replicas of tablet 1227f54d6ca8499d860df9d4103abf6d
I20260925 14:55:39.114542 17715 tablet_service.cc:1514] Processing DeleteTablet for tablet 1227f54d6ca8499d860df9d4103abf6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:37622
I20260925 14:55:39.114629 17832 tablet_replica.cc:330] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: stopping tablet replica
I20260925 14:55:39.114629 17435 tablet_service.cc:1514] Processing DeleteTablet for tablet 1227f54d6ca8499d860df9d4103abf6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:52606
I20260925 14:55:39.114681 17832 raft_consensus.cc:2229] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:39.114717 17832 raft_consensus.cc:2258] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:39.114727 17831 tablet_replica.cc:330] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: stopping tablet replica
I20260925 14:55:39.114771 17831 raft_consensus.cc:2229] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:55:39.114804 17831 raft_consensus.cc:2258] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:39.114943 17574 tablet_service.cc:1514] Processing DeleteTablet for tablet 1227f54d6ca8499d860df9d4103abf6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:55:39 UTC) from {username='oryx', principal='oryx/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:50286
I20260925 14:55:39.115029 17833 tablet_replica.cc:330] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: stopping tablet replica
I20260925 14:55:39.115073 17833 raft_consensus.cc:2229] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:55:39.115167 17833 raft_consensus.cc:2258] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:55:39.115489 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 17370
I20260925 14:55:39.115567 17831 ts_tablet_manager.cc:1905] T 1227f54d6ca8499d860df9d4103abf6d P e99acd84aace4b078e6f751d87d3ca18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:39.120690 17833 ts_tablet_manager.cc:1905] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:39.120890 17832 ts_tablet_manager.cc:1905] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:55:39.122615 17833 ts_tablet_manager.cc:1918] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:39.122615 17832 ts_tablet_manager.cc:1918] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:55:39.122684 17832 log.cc:1196] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/1227f54d6ca8499d860df9d4103abf6d
I20260925 14:55:39.122684 17833 log.cc:1196] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/1227f54d6ca8499d860df9d4103abf6d
I20260925 14:55:39.122949 17833 ts_tablet_manager.cc:1939] T 1227f54d6ca8499d860df9d4103abf6d P 7ed9e20d1ce642ef80ba45e3013a43b0: Deleting consensus metadata
I20260925 14:55:39.122996 17832 ts_tablet_manager.cc:1939] T 1227f54d6ca8499d860df9d4103abf6d P 9e8247c1bf4c48129e1bbb5ac1aa6e21: Deleting consensus metadata
I20260925 14:55:39.123332 17282 catalog_manager.cc:4841] TS 9e8247c1bf4c48129e1bbb5ac1aa6e21 (127.9.141.67:39861): tablet 1227f54d6ca8499d860df9d4103abf6d (table test-table [id=9e0adc77ebe446f485f6b2bcdfdc0c5a]) successfully deleted
I20260925 14:55:39.123332 17284 catalog_manager.cc:4841] TS 7ed9e20d1ce642ef80ba45e3013a43b0 (127.9.141.66:32775): tablet 1227f54d6ca8499d860df9d4103abf6d (table test-table [id=9e0adc77ebe446f485f6b2bcdfdc0c5a]) successfully deleted
W20260925 14:55:39.125128 17284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:45437 (error 108)
I20260925 14:55:39.125442 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 17510
I20260925 14:55:39.126354 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:39.125530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45437 (local address 127.0.0.1:52638)
0925 14:55:39.126277 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:39.126318 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:45437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":45,"threads_started":1}
W20260925 14:55:39.126530 17284 catalog_manager.cc:4584] TS e99acd84aace4b078e6f751d87d3ca18 (127.9.141.65:45437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1227f54d6ca8499d860df9d4103abf6d: Network error: Client connection negotiation failed: client connection to 127.9.141.65:45437: connect: Connection refused (error 111)
I20260925 14:55:39.131922 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 17650
I20260925 14:55:39.137180 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 17266
2026-09-25T14:55:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2756 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:39 dist-test-slave-1sv7 krb5kdc[17901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:39 dist-test-slave-1sv7 krb5kdc[17901](info): set up 2 sockets
Sep 25 14:55:39 dist-test-slave-1sv7 krb5kdc[17901](info): commencing operation
krb5kdc: starting...
W20260925 14:55:40.185065 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:40 dist-test-slave-1sv7 krb5kdc[17901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:40.307979 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:34505
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:34505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:40.389737 17914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:40.389942 17914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:40.389969 17914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:40.391170 17914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:40.391211 17914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:40.391227 17914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:40.391238 17914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:40.391250 17914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:40.392903 17914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:34505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:34505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:40.393149 17914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:40.393399 17914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:40.398284 17919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:40.398284 17920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:40.398274 17922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:40.398545 17914 server_base.cc:1034] running on GCE node
I20260925 14:55:40.398789 17914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:40.399093 17914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:40.400267 17914 hybrid_clock.cc:648] HybridClock initialized: now 1790348140400255 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:55:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1267 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:40 dist-test-slave-1sv7 krb5kdc[17927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:40 dist-test-slave-1sv7 krb5kdc[17927](info): set up 2 sockets
Sep 25 14:55:40 dist-test-slave-1sv7 krb5kdc[17927](info): commencing operation
krb5kdc: starting...
W20260925 14:55:41.445012 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:55:41 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:55:41.575553 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35229
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:41.654480 17940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:41.654702 17940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:41.654737 17940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:41.655874 17940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:41.655942 17940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:41.655967 17940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:41.655995 17940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:41.656019 17940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:41.657506 17940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:35229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:41.657764 17940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:41.657984 17940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:55:41.662871 17940 server_base.cc:1034] running on GCE node
W20260925 14:55:41.662930 17946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:41.662940 17945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:41.662981 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:41.663322 17940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:41.663602 17940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:41.664752 17940 hybrid_clock.cc:648] HybridClock initialized: now 1790348141664703 us; error 64 us; skew 500 ppm
Sep 25 14:55:41 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:41.667445 17940 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:41.667909 17940 webserver.cc:460] Webserver started at http://127.9.141.126:40803/ using document root <none> and password file <none>
I20260925 14:55:41.668169 17940 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:41.668216 17940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:41.668372 17940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:41.669535 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "a018f99775fdffa1d5e08c4f53013e85"
server_key_iv: "a1b23327ee867b89e6e89c80adbdf566"
server_key_version: "encryptionkey@0"
I20260925 14:55:41.669925 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "a018f99775fdffa1d5e08c4f53013e85"
server_key_iv: "a1b23327ee867b89e6e89c80adbdf566"
server_key_version: "encryptionkey@0"
I20260925 14:55:41.671330 17940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:41.672129 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:41.672305 17940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:41.672432 17940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "a018f99775fdffa1d5e08c4f53013e85"
server_key_iv: "a1b23327ee867b89e6e89c80adbdf566"
server_key_version: "encryptionkey@0"
I20260925 14:55:41.672518 17940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:41.686648 17940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:41.688627 17940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:41.688795 17940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:41.693030 17940 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:35229
I20260925 14:55:41.693060 18007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:35229 every 8 connection(s)
I20260925 14:55:41.693473 17940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:41.694372 18008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:41.696600 18008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Bootstrap starting.
I20260925 14:55:41.697162 18008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:41.697443 18008 log.cc:824] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:41.698140 18008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: No bootstrap required, opened a new log
I20260925 14:55:41.699676 18008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:41.699810 18008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:41.699848 18008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd13626f6d14b5e9f2d6cbe91b6c677, State: Initialized, Role: FOLLOWER
I20260925 14:55:41.699980 18008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:41.700054 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 17940
I20260925 14:55:41.700093 18008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:41.700136 18008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:41.700186 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:41.700193 18008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:41.700445 9781 external_mini_cluster.cc:1428] Setting key 8a32d3bd5fd7d58bffcaa665792b14af
I20260925 14:55:41.700830 18008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:41.700947 18008 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd13626f6d14b5e9f2d6cbe91b6c677; no voters:
I20260925 14:55:41.701107 18008 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:41.701239 18012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:41.701335 18008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:41.701413 18012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd13626f6d14b5e9f2d6cbe91b6c677, State: Running, Role: LEADER
I20260925 14:55:41.701506 18012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:41.701896 18012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd13626f6d14b5e9f2d6cbe91b6c677. Latest consensus state: current_term: 1 leader_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } } }
I20260925 14:55:41.701906 18014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } } }
I20260925 14:55:41.701994 18012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:41.702003 18014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:41.703044 18026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:41.703552 18026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:41.705248 18026 catalog_manager.cc:1347] Generated new cluster ID: 83bc28805be94de5b0009e4f2527dfd5
I20260925 14:55:41.705307 18026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:55:41 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:41.713089 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.700981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.9.141.126:35229)
0925 14:55:41.701186 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:41.701197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:41.701241 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:55:41.701376 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:41.701385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:41.701449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:41.702176 (+ 727us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:41.702527 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.704177 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.705458 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.706065 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.706202 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:41.709123 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:41.709128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:41.709129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:41.709141 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:41.711284 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.711709 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.711713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.711713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.711831 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.712055 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.712055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.712056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.712410 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:41.712584 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:41.712684 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:55:41.712831 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":99,"threads_started":1}
I20260925 14:55:41.719815 18026 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:41.720543 18026 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:41.728252 18026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Generated new TSK 0
I20260925 14:55:41.728470 18026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:55:41.757404 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35229
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:41.840961 18034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:41.841181 18034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:41.841204 18034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:41.842423 18034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:41.842473 18034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:41.842500 18034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:41.843933 18034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:41.844172 18034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:41.844478 18034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:41.849519 18042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:41.849558 18039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:41.849519 18040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:41.849867 18034 server_base.cc:1034] running on GCE node
I20260925 14:55:41.850087 18034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:41.850292 18034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:41.851438 18034 hybrid_clock.cc:648] HybridClock initialized: now 1790348141851419 us; error 42 us; skew 500 ppm
Sep 25 14:55:41 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:41.854085 18034 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:55:41.854566 18034 webserver.cc:460] Webserver started at http://127.9.141.65:35133/ using document root <none> and password file <none>
I20260925 14:55:41.854801 18034 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:41.854873 18034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:41.855026 18034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:41.856060 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "10e17f4f0c4f46b0a2d8467526be6f8f"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "2b80c7a1b9ad1a31cf8e0c74368752c1"
server_key_iv: "8a0ea0a2bdea2a5369cc5f7e5d73bfc6"
server_key_version: "encryptionkey@0"
I20260925 14:55:41.856518 18034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "10e17f4f0c4f46b0a2d8467526be6f8f"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "2b80c7a1b9ad1a31cf8e0c74368752c1"
server_key_iv: "8a0ea0a2bdea2a5369cc5f7e5d73bfc6"
server_key_version: "encryptionkey@0"
I20260925 14:55:41.857892 18034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:41.858664 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:41.858853 18034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:41.858951 18034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "10e17f4f0c4f46b0a2d8467526be6f8f"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "2b80c7a1b9ad1a31cf8e0c74368752c1"
server_key_iv: "8a0ea0a2bdea2a5369cc5f7e5d73bfc6"
server_key_version: "encryptionkey@0"
I20260925 14:55:41.859031 18034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:41.884138 18034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:41.886884 18034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:41.887033 18034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:41.887269 18034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:41.887580 18034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:41.887611 18034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:41.887639 18034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:41.887657 18034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:41.895819 18034 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:42635
I20260925 14:55:41.895902 18162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:42635 every 8 connection(s)
I20260925 14:55:41.896234 18034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:41.902521 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18034
I20260925 14:55:41.902626 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:41.902848 9781 external_mini_cluster.cc:1428] Setting key 01aaed8b9387301be5a4265e1cad78eb
Sep 25 14:55:41 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348141, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:41.906958 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.897131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:41639 (local address 127.9.141.126:35229)
0925 14:55:41.897199 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:41.897201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:41.897430 (+ 229us) server_negotiation.cc:407] Connection header received
0925 14:55:41.897951 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:41.897952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:41.897965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:41.898024 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:41.898968 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.899486 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.900549 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.900750 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.900802 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:41.903543 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:41.903544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:41.903545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:41.903552 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:41.905310 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.905837 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.905838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.905838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.905891 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:41.906263 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:41.906264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:41.906265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:41.906381 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:41.906444 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:41.906812 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 14:55:41.906884 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:55:41.907234 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:41.897121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:41639)
0925 14:55:41.897305 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:41.897342 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:41.897845 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:41.898105 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:41.898108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:41.898689 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:41.898878 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:41.898882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.899587 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.899588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:41.900455 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:41.900458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:41.900851 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:41.900852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:41.900995 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:41.901613 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:41.901623 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:41.903441 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:41.905408 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:41.905416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:41.905418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:41.905753 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:41.905960 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:41.905961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:41.905961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:41.906106 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:41.906461 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:41.906468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:41.906658 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 14:55:41.906780 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 14:55:41.906959 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260925 14:55:41.907820 18163 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35229
I20260925 14:55:41.907936 18163 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:41.908179 18163 heartbeater.cc:511] Master 127.9.141.126:35229 requested a full tablet report, sending...
I20260925 14:55:41.908722 17972 ts_manager.cc:194] Registered new tserver with Master: 10e17f4f0c4f46b0a2d8467526be6f8f (127.9.141.65:42635)
I20260925 14:55:41.909370 17972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:41639
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:55:41.932317 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35229
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:42.013840 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:42.014036 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:42.014060 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:42.015232 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:42.015276 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:42.015303 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:42.016768 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:42.017010 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:42.017241 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:42.022207 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.022256 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.022487 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.022560 18168 server_base.cc:1034] running on GCE node
I20260925 14:55:42.022792 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:42.023044 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:42.024219 18168 hybrid_clock.cc:648] HybridClock initialized: now 1790348142024190 us; error 46 us; skew 500 ppm
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:42.027124 18168 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:55:42.027591 18168 webserver.cc:460] Webserver started at http://127.9.141.66:37553/ using document root <none> and password file <none>
I20260925 14:55:42.027860 18168 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:42.027907 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:42.028064 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:42.029191 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "2a360d6b37964bc392748ea37056b700"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "50d9d00e84f075276ad607648846b989"
server_key_iv: "248d26e2ad98e38060e9d219618acfd9"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.029551 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "2a360d6b37964bc392748ea37056b700"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "50d9d00e84f075276ad607648846b989"
server_key_iv: "248d26e2ad98e38060e9d219618acfd9"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.030864 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:55:42.031608 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.031811 18168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:42.031877 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "2a360d6b37964bc392748ea37056b700"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "50d9d00e84f075276ad607648846b989"
server_key_iv: "248d26e2ad98e38060e9d219618acfd9"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.031973 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:42.044569 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:42.046648 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:42.046905 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:42.047116 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:42.047510 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:42.047546 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.047567 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:42.047616 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.053145 18168 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:44265
I20260925 14:55:42.053226 18296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:44265 every 8 connection(s)
I20260925 14:55:42.053522 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:42.057382 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18168
I20260925 14:55:42.057484 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:42.057677 9781 external_mini_cluster.cc:1428] Setting key 7af3fa24aeda5f0d40fc2d4ea26c93a3
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:42.063499 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.054272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:47345 (local address 127.9.141.126:35229)
0925 14:55:42.054326 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.054328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.054569 (+ 241us) server_negotiation.cc:407] Connection header received
0925 14:55:42.055013 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.055014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.055024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.055133 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.056033 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.056602 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.057826 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.058128 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.058194 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.060640 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:42.060642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:42.060642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:42.060650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:42.062095 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:42.062524 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:42.062526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:42.062526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:42.062583 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:42.062872 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:42.062872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:42.062873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:42.062982 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:42.063057 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:42.063324 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 14:55:42.063412 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:42.063694 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.054266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:47345)
0925 14:55:42.054430 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.054464 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.054929 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.055189 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.055192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.055752 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.055935 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.055939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.056680 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.056681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.057706 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.057709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.058205 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.058206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.058429 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.058934 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.058945 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.060547 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:42.062194 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:42.062200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:42.062202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:42.062427 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:42.062664 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:42.062665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:42.062665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:42.062794 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:42.063050 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:42.063055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:42.063192 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:55:42.063304 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:55:42.063454 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260925 14:55:42.064145 18297 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35229
I20260925 14:55:42.064239 18297 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:42.064459 18297 heartbeater.cc:511] Master 127.9.141.126:35229 requested a full tablet report, sending...
I20260925 14:55:42.064864 17972 ts_manager.cc:194] Registered new tserver with Master: 2a360d6b37964bc392748ea37056b700 (127.9.141.66:44265)
I20260925 14:55:42.065392 17972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:47345
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:55:42.085246 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35229
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:42.165647 18302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:42.165855 18302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:42.165879 18302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:42.166989 18302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:42.167023 18302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:42.167049 18302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:42.168416 18302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:42.168589 18302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:42.168759 18302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:42.173892 18310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.173904 18307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.173933 18308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.174263 18302 server_base.cc:1034] running on GCE node
I20260925 14:55:42.174499 18302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:42.174748 18302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:42.175921 18302 hybrid_clock.cc:648] HybridClock initialized: now 1790348142175903 us; error 32 us; skew 500 ppm
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:42.178366 18302 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:55:42.178735 18302 webserver.cc:460] Webserver started at http://127.9.141.67:37613/ using document root <none> and password file <none>
I20260925 14:55:42.178953 18302 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:42.179010 18302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:42.179177 18302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:42.180200 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "68d821b5c4a044f98e3adeedbbe7068b"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "be25987c10005c1ce05f36c3963dacea"
server_key_iv: "20b3c6a1b1d45da1400e87a3cbb637c4"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.180581 18302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "68d821b5c4a044f98e3adeedbbe7068b"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "be25987c10005c1ce05f36c3963dacea"
server_key_iv: "20b3c6a1b1d45da1400e87a3cbb637c4"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.181874 18302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:42.182600 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.182765 18302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.182891 18302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "68d821b5c4a044f98e3adeedbbe7068b"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "be25987c10005c1ce05f36c3963dacea"
server_key_iv: "20b3c6a1b1d45da1400e87a3cbb637c4"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.182968 18302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:42.200160 18302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:42.202438 18302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:42.202593 18302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:42.202859 18302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:42.203220 18302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:42.203274 18302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.203310 18302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:42.203336 18302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.209172 18302 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:37369
I20260925 14:55:42.209214 18430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:37369 every 8 connection(s)
I20260925 14:55:42.209533 18302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:42.210022 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18302
I20260925 14:55:42.210141 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:42.210362 9781 external_mini_cluster.cc:1428] Setting key 940fb2563a2a7636ca751ce9bc1786c0
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:55:42.219018 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.210201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:43053 (local address 127.9.141.126:35229)
0925 14:55:42.210253 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.210256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.210628 (+ 372us) server_negotiation.cc:407] Connection header received
0925 14:55:42.211188 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.211189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.211200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.211262 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.212583 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.213217 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.214188 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.214369 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.214419 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.216472 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:42.216474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:42.216474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:42.216480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:42.217867 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:42.218224 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:42.218225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:42.218225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:42.218272 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:42.218478 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:42.218479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:42.218479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:42.218571 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:42.218620 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:42.218899 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:55:42.218957 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:55:42.219249 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.210185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:43053)
0925 14:55:42.210372 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.210425 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.211090 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.211348 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.211353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.212191 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.212499 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.212504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.213329 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.213330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.214115 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.214117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.214440 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.214440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.214538 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.214967 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.214975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.216330 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:42.217943 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:42.217948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:42.217950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:42.218160 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:42.218331 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:55:42.218332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:55:42.218332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:55:42.218426 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:55:42.218632 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:42.218639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:42.218784 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:55:42.218874 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:55:42.218988 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 14:55:42.219803 18431 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35229
I20260925 14:55:42.219899 18431 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:42.220083 18431 heartbeater.cc:511] Master 127.9.141.126:35229 requested a full tablet report, sending...
I20260925 14:55:42.220474 17972 ts_manager.cc:194] Registered new tserver with Master: 68d821b5c4a044f98e3adeedbbe7068b (127.9.141.67:37369)
I20260925 14:55:42.220794 17972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:43053
I20260925 14:55:42.222237 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab.
Entry for principal oryx/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab.
WARNING: no policy specified for oryx/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab.
Entry for principal oryx/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab.
WARNING: no policy specified for oryx/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab.
Entry for principal oryx/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab.
I20260925 14:55:42.320995 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18034
I20260925 14:55:42.325380 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18168
I20260925 14:55:42.329726 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18302
I20260925 14:55:42.334100 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 17940
I20260925 14:55:42.339324 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35229
--webserver_interface=127.9.141.126
--webserver_port=40803
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:35229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:55:42.420179 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:42.420414 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:42.420439 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:42.421614 18440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:42.421658 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:42.421674 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:42.421686 18440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:42.421698 18440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:42.423167 18440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:35229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=40803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:42.423421 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:42.423672 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:42.428886 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.428943 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.428916 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.429183 18440 server_base.cc:1034] running on GCE node
I20260925 14:55:42.429399 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:42.429644 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:42.430821 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790348142430809 us; error 38 us; skew 500 ppm
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:42.433470 18440 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:55:42.433964 18440 webserver.cc:460] Webserver started at http://127.9.141.126:40803/ using document root <none> and password file <none>
I20260925 14:55:42.434212 18440 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:42.434297 18440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:42.435648 18440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.436430 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.436607 18440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.436695 18440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "a018f99775fdffa1d5e08c4f53013e85"
server_key_iv: "a1b23327ee867b89e6e89c80adbdf566"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.437047 18440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:42.467252 18440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:42.469446 18440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:42.469626 18440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:42.474140 18440 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:35229
I20260925 14:55:42.474196 18507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:35229 every 8 connection(s)
I20260925 14:55:42.474597 18440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:42.476163 18508 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:55:42.476907 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Bootstrap starting.
I20260925 14:55:42.484527 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18440
I20260925 14:55:42.484987 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:42635
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:35229
--webserver_port=35133
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:55:42.485436 18508 log.cc:824] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:42.487485 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:55:42.487799 18508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Bootstrap complete.
I20260925 14:55:42.490387 18508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:42.490634 18508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd13626f6d14b5e9f2d6cbe91b6c677, State: Initialized, Role: FOLLOWER
I20260925 14:55:42.490824 18508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:42.490906 18508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:42.490960 18508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:42.491014 18508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:55:42.492080 18508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:42.492170 18508 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd13626f6d14b5e9f2d6cbe91b6c677; no voters:
I20260925 14:55:42.492421 18508 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:55:42.492456 18512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:55:42.492584 18512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [term 2 LEADER]: Becoming Leader. State: Replica: 3bd13626f6d14b5e9f2d6cbe91b6c677, State: Running, Role: LEADER
I20260925 14:55:42.492677 18512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } }
I20260925 14:55:42.492712 18508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:42.492992 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd13626f6d14b5e9f2d6cbe91b6c677. Latest consensus state: current_term: 2 leader_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } } }
I20260925 14:55:42.493064 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:42.493278 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd13626f6d14b5e9f2d6cbe91b6c677" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35229 } } }
I20260925 14:55:42.493357 18516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:42.493369 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:42.493932 18516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:42.494098 18516 catalog_manager.cc:1259] Loaded cluster ID: 83bc28805be94de5b0009e4f2527dfd5
I20260925 14:55:42.494395 18516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:42.495373 18516 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:42.495577 18516 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bd13626f6d14b5e9f2d6cbe91b6c677: Loaded TSK: 0
I20260925 14:55:42.495820 18516 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:55:42.578037 18511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:42.578294 18511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:42.578344 18511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:42.579581 18511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:42.579651 18511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:42.579702 18511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:42.581218 18511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.65.keytab
--principal=oryx/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:42635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=35133
--tserver_master_addrs=127.9.141.126:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:42.581496 18511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:42.581739 18511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:42.587778 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.587833 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.587821 18511 server_base.cc:1034] running on GCE node
W20260925 14:55:42.587793 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.588201 18511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:42.588531 18511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:42.589699 18511 hybrid_clock.cc:648] HybridClock initialized: now 1790348142589671 us; error 44 us; skew 500 ppm
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:42.592444 18511 init.cc:377] Logged in from keytab as oryx/127.9.141.65@KRBTEST.COM (short username oryx)
I20260925 14:55:42.592885 18511 webserver.cc:460] Webserver started at http://127.9.141.65:35133/ using document root <none> and password file <none>
I20260925 14:55:42.593111 18511 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:42.593190 18511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:42.594738 18511 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.595461 18544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.595634 18511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.595731 18511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "10e17f4f0c4f46b0a2d8467526be6f8f"
format_stamp: "Formatted at 2026-09-25 14:55:41 on dist-test-slave-1sv7"
server_key: "2b80c7a1b9ad1a31cf8e0c74368752c1"
server_key_iv: "8a0ea0a2bdea2a5369cc5f7e5d73bfc6"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.596109 18511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:42.623050 18511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:42.626675 18511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:42.626986 18511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:42.627293 18511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:42.627733 18511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:42.627791 18511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.627830 18511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:42.627857 18511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.635107 18511 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:42635
I20260925 14:55:42.635145 18657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:42635 every 8 connection(s)
I20260925 14:55:42.635496 18511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:42.640810 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18511
I20260925 14:55:42.641115 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:44265
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:35229
--webserver_port=37553
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.644996 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.636280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:37625)
0925 14:55:42.636463 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.636510 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.636984 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.637452 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.637456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.638035 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.638223 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.638227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.639351 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.639352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.640512 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.640514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.641205 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.641206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.641365 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.641905 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.641916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.644779 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
W20260925 14:55:42.645057 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.636327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:37625 (local address 127.9.141.126:35229)
0925 14:55:42.636526 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.636536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.636597 (+ 61us) server_negotiation.cc:407] Connection header received
0925 14:55:42.637162 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.637167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.637209 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.637322 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.638337 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.639252 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.640616 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.641110 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.641230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.644899 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:37625: BlockingRecv error: recv got EOF from 127.9.141.65:37625 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
W20260925 14:55:42.645455 18658 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:35229 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.141.126:35229: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.650442 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.645706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:46647)
0925 14:55:42.645757 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.645765 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.645814 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.645958 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.645959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.646100 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.646143 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.646145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.646777 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.646778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.647278 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.647279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.647629 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.647630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.647700 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.648211 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.648215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.650336 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 14:55:42.650703 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.645768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:46647 (local address 127.9.141.126:35229)
0925 14:55:42.645831 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.645833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.645841 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:42.645881 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.645882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.645891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.645955 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.646222 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.646686 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.647371 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.647560 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.647632 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.650621 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:46647: BlockingRecv error: recv got EOF from 127.9.141.65:46647 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.655701 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.650790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:53477)
0925 14:55:42.650900 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.650912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.650960 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.651115 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.651116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.651245 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.651272 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.651274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.652325 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.652326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.652867 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.652869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.653373 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.653374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.653409 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.653863 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.653867 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.655657 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 14:55:42.655827 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.650853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:53477 (local address 127.9.141.126:35229)
0925 14:55:42.650903 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.650905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.650948 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:55:42.651017 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.651018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.651031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.651107 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.651349 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.652222 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.652955 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.653294 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.653371 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.655762 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:53477: BlockingRecv error: recv got EOF from 127.9.141.65:53477 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:55:42.655920 18658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:55:42.730520 18662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:42.730751 18662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:42.730790 18662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:42.731962 18662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:42.732038 18662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:42.732085 18662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:42.733563 18662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.66.keytab
--principal=oryx/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:44265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=37553
--tserver_master_addrs=127.9.141.126:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:42.733819 18662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:42.734047 18662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:55:42.739306 18662 server_base.cc:1034] running on GCE node
W20260925 14:55:42.739295 18667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.739265 18670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.739277 18668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.739729 18662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:42.739925 18662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:42.741103 18662 hybrid_clock.cc:648] HybridClock initialized: now 1790348142741091 us; error 34 us; skew 500 ppm
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:42.748508 18662 init.cc:377] Logged in from keytab as oryx/127.9.141.66@KRBTEST.COM (short username oryx)
I20260925 14:55:42.748987 18662 webserver.cc:460] Webserver started at http://127.9.141.66:37553/ using document root <none> and password file <none>
I20260925 14:55:42.749254 18662 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:42.749339 18662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:42.750705 18662 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.751271 18677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.751431 18662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:42.751531 18662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "2a360d6b37964bc392748ea37056b700"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "50d9d00e84f075276ad607648846b989"
server_key_iv: "248d26e2ad98e38060e9d219618acfd9"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.751909 18662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:42.765947 18662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:42.767930 18662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:42.768126 18662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:42.768458 18662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:42.768846 18662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:42.768908 18662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.768951 18662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:42.768977 18662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.774642 18662 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:44265
I20260925 14:55:42.774730 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:44265 every 8 connection(s)
I20260925 14:55:42.775075 18662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:42.776852 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18662
I20260925 14:55:42.777141 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:37369
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:35229
--webserver_port=37613
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:55:42.783679 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.775817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:52841 (local address 127.9.141.126:35229)
0925 14:55:42.775870 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.775873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.776129 (+ 256us) server_negotiation.cc:407] Connection header received
0925 14:55:42.776687 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.776707 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.776722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.776817 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.777918 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.778430 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.779524 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.779759 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.779819 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.783631 (+ 3812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:52841: BlockingRecv error: recv got EOF from 127.9.141.66:52841 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:55:42.783787 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.775811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:52841)
0925 14:55:42.775982 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.776016 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.776568 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.776906 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.776909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.777622 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.777810 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.777815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.778534 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.778536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.779439 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.779442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.779847 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.779848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.779994 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.780744 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.780755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.783539 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
W20260925 14:55:42.783991 18791 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:35229 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.141.126:35229: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.789108 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.784233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:52483)
0925 14:55:42.784280 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.784290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.784326 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.784498 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.784499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.784667 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.784721 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.784723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.785320 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.785321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.786233 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.786235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.786585 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.786587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.786662 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.787073 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.787079 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.789038 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:55:42.789204 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.784288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:52483 (local address 127.9.141.126:35229)
0925 14:55:42.784324 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.784326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.784335 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:42.784371 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.784372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.784414 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.784477 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.784801 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.785242 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.786324 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.786499 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.786555 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.789122 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:52483: BlockingRecv error: recv got EOF from 127.9.141.66:52483 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.793658 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.789328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:47269)
0925 14:55:42.789379 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.789391 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.789431 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.789550 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.789551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.789654 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.789680 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.789681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.790559 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.790560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.791031 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.791032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.791525 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.791526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.791569 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.792009 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.792013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.793594 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:55:42.793789 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.789411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:47269 (local address 127.9.141.126:35229)
0925 14:55:42.789444 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.789446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.789453 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:55:42.789477 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.789477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.789486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.789539 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.789749 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.790469 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.791104 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.791400 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.791465 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.793736 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:47269: BlockingRecv error: recv got EOF from 127.9.141.66:47269 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:55:42.793845 18791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:55:42.862635 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:42.862833 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:42.862854 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:42.863925 18794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:42.863958 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:42.863984 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:42.865360 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/krb5kdc/oryx_127.9.141.67.keytab
--principal=oryx/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:37369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=37613
--tserver_master_addrs=127.9.141.126:35229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:42.865561 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:42.865728 18794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:42.870714 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.870733 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:42.870795 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:42.871080 18794 server_base.cc:1034] running on GCE node
I20260925 14:55:42.871274 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:42.871475 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:42.872702 18794 hybrid_clock.cc:648] HybridClock initialized: now 1790348142872690 us; error 42 us; skew 500 ppm
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348142, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:55:42.875308 18794 init.cc:377] Logged in from keytab as oryx/127.9.141.67@KRBTEST.COM (short username oryx)
I20260925 14:55:42.875833 18794 webserver.cc:460] Webserver started at http://127.9.141.67:37613/ using document root <none> and password file <none>
I20260925 14:55:42.876082 18794 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:42.876163 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:42.877651 18794 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.878334 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.878533 18794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:42.878630 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "68d821b5c4a044f98e3adeedbbe7068b"
format_stamp: "Formatted at 2026-09-25 14:55:42 on dist-test-slave-1sv7"
server_key: "be25987c10005c1ce05f36c3963dacea"
server_key_iv: "20b3c6a1b1d45da1400e87a3cbb637c4"
server_key_version: "encryptionkey@0"
I20260925 14:55:42.879029 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:42.889719 18794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:42.892262 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:42.892445 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:42.892679 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:42.893044 18794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:42.893095 18794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.893160 18794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:42.893193 18794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:42.899466 18794 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:37369
I20260925 14:55:42.899507 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:37369 every 8 connection(s)
I20260925 14:55:42.899834 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:42.902410 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18794
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.908245 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.900578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:58299)
0925 14:55:42.900755 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.900790 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.901298 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.901547 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.901551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.902124 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.902323 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.902327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.903210 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.903212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.904299 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.904302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.904740 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.904741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.904874 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.905407 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.905423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.908097 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
W20260925 14:55:42.908254 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.900583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:58299 (local address 127.9.141.126:35229)
0925 14:55:42.900652 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.900654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.900872 (+ 218us) server_negotiation.cc:407] Connection header received
0925 14:55:42.901415 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.901417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.901429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.901478 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.902435 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.903086 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.904413 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.904630 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.904730 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.908198 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:58299: BlockingRecv error: recv got EOF from 127.9.141.67:58299 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 14:55:42.908516 18923 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:35229 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.141.126:35229: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
I20260925 14:55:42.910347 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.902975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.9.141.126:35229)
0925 14:55:42.903148 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.903149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.903157 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:42.903176 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.903177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.903187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.903271 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.903853 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.904332 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.905138 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.905348 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.905441 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.906839 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:42.906843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:42.906843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:55:42.906852 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:42.908866 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:42.909247 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:42.909268 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:42.909269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:42.909383 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:55:42.909580 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:55:42.909581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:55:42.909582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:55:42.909855 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:42.909993 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:42.910202 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:55:42.910286 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.913506 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.908772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:58759)
0925 14:55:42.908829 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.908841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.908883 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.909000 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.909001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.909160 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.909217 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.909219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.910093 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.910094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.910860 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.910863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.911346 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.911347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.911408 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.911817 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.911822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.913463 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:55:42.913724 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.908825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:58759 (local address 127.9.141.126:35229)
0925 14:55:42.908887 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.908889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.908914 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:55:42.908934 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.908935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.908944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.908985 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.909286 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.910003 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.910950 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.911245 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.911323 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.913644 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:58759: BlockingRecv error: recv got EOF from 127.9.141.67:58759 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:42 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:42.917964 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:42.913820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:46783)
0925 14:55:42.913869 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:42.913878 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:42.913943 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:42.914095 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:42.914096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:42.914224 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:42.914249 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.914251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.914850 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.914851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.915346 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:42.915348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.915717 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.915718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:42.915770 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.916143 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:42.916147 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:42.917923 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:55:42.918128 18661 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:42.913884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:46783 (local address 127.9.141.126:35229)
0925 14:55:42.913940 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:42.913942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:42.913951 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:42.914012 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:42.914013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:42.914021 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:42.914092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:42.914337 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.914772 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.915430 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:42.915618 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:42.915718 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:42.918059 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:46783: BlockingRecv error: recv got EOF from 127.9.141.67:46783 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 14:55:42.918151 18923 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:55:43 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:43 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:43.660763 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:43.656277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:36143)
0925 14:55:43.656428 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:43.656440 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:43.656498 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:43.656627 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:43.656628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:43.656787 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:43.656867 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:43.656869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.657480 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.657480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:43.658052 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:43.658054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.658396 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.658396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:43.658440 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.658872 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:43.658877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:43.660695 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
W20260925 14:55:43.660837 18929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:43.656342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:36143 (local address 127.9.141.126:35229)
0925 14:55:43.656491 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:43.656493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:43.656501 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:43.656541 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:43.656541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:43.656550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:43.656609 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:43.656930 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.657415 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.658122 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.658309 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.658361 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.660786 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:36143: BlockingRecv error: recv got EOF from 127.9.141.65:36143 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
Sep 25 14:55:43 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:43 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:43.799373 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:43.794247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:39001)
0925 14:55:43.794377 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:43.794388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:43.794437 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:43.794609 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:43.794609 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:43.794763 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:43.794810 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:43.794812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.795538 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.795539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:43.796081 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:43.796082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.796516 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.796518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:43.796565 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.797178 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:43.797182 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:43.799308 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260925 14:55:43.799515 18929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:43.794346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:39001 (local address 127.9.141.126:35229)
0925 14:55:43.794408 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:43.794412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:43.794428 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:55:43.794504 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:43.794505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:43.794515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:43.794572 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:43.794906 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.795457 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.796170 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.796416 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.796489 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.799439 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:39001: BlockingRecv error: recv got EOF from 127.9.141.66:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 14:55:43 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:43 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:43.928179 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:43.918638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:44381)
0925 14:55:43.918765 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:43.918777 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:43.918840 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:43.919055 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:43.919056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:43.919280 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:43.919327 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:43.919329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.919976 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.919977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:43.920594 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:43.920596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.920970 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.920970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:43.921017 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.926281 (+ 5264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:43.926286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:43.928101 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
W20260925 14:55:43.928331 18929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:43.918682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:44381 (local address 127.9.141.126:35229)
0925 14:55:43.918754 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:43.918756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:43.918849 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:55:43.918954 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:43.918955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:43.918970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:43.919032 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:43.919396 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.919888 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.920696 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:43.920884 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:43.920949 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:43.928233 (+ 7284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:44381: BlockingRecv error: recv got EOF from 127.9.141.67:44381 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:55:44 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:44 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:44.666360 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.661303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:34703)
0925 14:55:44.661470 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:44.661481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:44.661537 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:44.661665 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:44.661666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:44.661827 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:44.661877 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.661878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.662626 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.662627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.663343 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.663346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.663822 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.663823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.663882 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.664417 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:44.664422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:44.666297 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
W20260925 14:55:44.666477 18933 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:44.661413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:34703 (local address 127.9.141.126:35229)
0925 14:55:44.661539 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:44.661541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:44.661559 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:55:44.661580 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:44.661581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:44.661590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:44.661648 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:44.661952 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.662540 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.663445 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.663656 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.663719 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.666389 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:34703: BlockingRecv error: recv got EOF from 127.9.141.65:34703 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
Sep 25 14:55:44 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:44 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:44.804638 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.799911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:52401)
0925 14:55:44.800077 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:44.800087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:44.800151 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:44.800278 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:44.800279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:44.800452 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:44.800501 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.800502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.801126 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.801127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.801671 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.801672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.802008 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.802010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.802059 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.802490 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:44.802495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:44.804543 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
W20260925 14:55:44.804687 18933 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:44.800011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:52401 (local address 127.9.141.126:35229)
0925 14:55:44.800059 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:44.800061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:44.800152 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:55:44.800195 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:44.800196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:44.800208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:44.800257 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:44.800568 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.801049 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.801745 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.801923 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.801975 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.804621 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:52401: BlockingRecv error: recv got EOF from 127.9.141.66:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 14:55:44 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:44 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:44.933485 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:44.928711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:51355)
0925 14:55:44.928886 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:44.928900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:44.928974 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:44.929152 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:44.929153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:44.929311 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:44.929361 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.929362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.930085 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.930086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.930627 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:44.930628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.930979 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.930980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:44.931017 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.931549 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:44.931553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:44.933429 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
W20260925 14:55:44.933632 18933 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:44.928872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:51355 (local address 127.9.141.126:35229)
0925 14:55:44.928943 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:44.928945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:44.928953 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:44.929032 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:44.929033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:44.929046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:44.929103 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:44.929444 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.930004 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.930713 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:44.930911 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:44.930970 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:44.933534 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:51355: BlockingRecv error: recv got EOF from 127.9.141.67:51355 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:55:45 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:45 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:45.671751 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.666947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:42431)
0925 14:55:45.667117 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.667128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.667182 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.667378 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.667379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.667544 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:45.667594 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.667595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.668425 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.668425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.668987 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.668989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.669367 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.669368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.669420 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.669855 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:45.669859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:45.671694 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
W20260925 14:55:45.671926 18937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:45.667075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:42431 (local address 127.9.141.126:35229)
0925 14:55:45.667214 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.667216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.667224 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:45.667247 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.667248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.667259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.667320 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.667684 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.668310 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.669063 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.669251 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.669305 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.671804 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:42431: BlockingRecv error: recv got EOF from 127.9.141.65:42431 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
Sep 25 14:55:45 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:45 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:45.809976 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.805122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:43825)
0925 14:55:45.805278 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.805291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.805387 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.805525 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.805526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.805697 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:45.805746 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.805748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.806493 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.806501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.807052 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.807055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.807419 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.807419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.807460 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.807946 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:45.807951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:45.809874 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
W20260925 14:55:45.810019 18937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:45.805213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:43825 (local address 127.9.141.126:35229)
0925 14:55:45.805277 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.805279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.805391 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:55:45.805436 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.805436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.805447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.805503 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.805846 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.806396 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.807151 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.807347 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.807404 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.809955 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:43825: BlockingRecv error: recv got EOF from 127.9.141.66:43825 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:55:45 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:45 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:45.939050 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:45.934073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:39919)
0925 14:55:45.934323 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:45.934334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:45.934405 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:45.934538 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:45.934539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:45.934699 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:45.934747 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.934749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.935410 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.935411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.936010 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:45.936012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.936357 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.936358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:45.936425 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.936968 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:45.936974 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:45.938937 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":197,"threads_started":1}
W20260925 14:55:45.939148 18937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:45.934198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:39919 (local address 127.9.141.126:35229)
0925 14:55:45.934259 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:45.934262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:45.934397 (+ 135us) server_negotiation.cc:407] Connection header received
0925 14:55:45.934463 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:45.934463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:45.934475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:45.934532 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:45.934839 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.935322 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.936096 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:45.936277 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:45.936332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:45.939048 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:39919: BlockingRecv error: recv got EOF from 127.9.141.67:39919 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 14:55:46 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:46 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:46.677129 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:46.672362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:39669)
0925 14:55:46.672510 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:46.672522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:46.672576 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:46.672796 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:46.672798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:46.672976 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:46.673025 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:46.673027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.673711 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.673712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:46.674251 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:46.674253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.674649 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.674650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:46.674698 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:46.675231 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:46.675236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:46.677034 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
W20260925 14:55:46.677192 18941 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:46.672445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:39669 (local address 127.9.141.126:35229)
0925 14:55:46.672569 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:46.672572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:46.672583 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:55:46.672656 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:46.672657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:46.672675 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:46.672783 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:46.673096 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.673613 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.674354 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.674568 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.674631 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:46.677119 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:39669: BlockingRecv error: recv got EOF from 127.9.141.65:39669 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
Sep 25 14:55:46 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:46 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:46.815529 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:46.810576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:53709)
0925 14:55:46.810715 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:46.810726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:46.810775 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:46.810991 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:46.810992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:46.811178 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:46.811225 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:46.811227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.811937 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.811938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:46.812535 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:46.812537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.812884 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.812885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:46.812941 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:46.813460 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:46.813465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:46.815434 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260925 14:55:46.815584 18941 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:46.810797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:53709 (local address 127.9.141.126:35229)
0925 14:55:46.810879 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:46.810881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:46.810887 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:55:46.810908 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:46.810908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:46.810918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:46.810984 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:46.811309 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.811849 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.812617 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.812811 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.812887 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:46.815523 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:53709: BlockingRecv error: recv got EOF from 127.9.141.66:53709 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 14:55:46 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:46 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:46.944738 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:46.939680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:54851)
0925 14:55:46.939857 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:46.939870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:46.939946 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:46.940128 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:46.940129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:46.940299 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:46.940347 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:46.940349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.941123 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.941124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:46.941699 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:46.941712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.942073 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.942073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:46.942117 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:46.942600 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:46.942605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:46.944629 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
W20260925 14:55:46.944813 18941 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:46.939740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:54851 (local address 127.9.141.126:35229)
0925 14:55:46.939811 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:46.939813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:46.939954 (+ 141us) server_negotiation.cc:407] Connection header received
0925 14:55:46.940014 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:46.940014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:46.940049 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:46.940091 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:46.940481 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.941041 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.941810 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:46.941995 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:46.942060 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:46.944741 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:54851: BlockingRecv error: recv got EOF from 127.9.141.67:54851 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 14:55:47 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:47 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:47.682605 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:47.677617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:37563)
0925 14:55:47.677764 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:47.677784 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:47.677841 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:47.677981 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:47.677982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:47.678140 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:47.678187 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:47.678188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.678887 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.678888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:47.679529 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:47.679531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.679895 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.679896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:47.679953 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:47.680569 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:47.680573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:47.682547 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
W20260925 14:55:47.682751 18945 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:47.677665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:37563 (local address 127.9.141.126:35229)
0925 14:55:47.677795 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:47.677797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:47.677826 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:55:47.677885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:47.677885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:47.677898 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:47.677963 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:47.678278 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.678795 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.679613 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.679802 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.679858 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:47.682649 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:37563: BlockingRecv error: recv got EOF from 127.9.141.65:37563 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
Sep 25 14:55:47 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:47 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:47.824960 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:47.816251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:55675)
0925 14:55:47.816440 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:47.816453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:47.816566 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:47.816730 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:47.816731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:47.816951 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:47.816997 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:47.817000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.817904 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.817905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:47.818438 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:47.818439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.818844 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.818845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:47.818916 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:47.823043 (+ 4127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:47.823048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:47.824852 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
W20260925 14:55:47.825003 18945 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:47.816518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:55675 (local address 127.9.141.126:35229)
0925 14:55:47.816588 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:47.816590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:47.816599 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:47.816630 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:47.816632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:47.816644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:47.816722 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:47.817092 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.817816 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.818529 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.818755 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.818843 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:47.824941 (+ 6098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:55675: BlockingRecv error: recv got EOF from 127.9.141.66:55675 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:55:47 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:47 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:47.950480 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:47.945443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:48223)
0925 14:55:47.945611 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:47.945624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:47.945699 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:47.945834 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:47.945834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:47.946001 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:47.946049 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:47.946051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.946712 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.946713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:47.947239 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:47.947240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.947611 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.947612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:47.947656 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:47.948325 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:47.948329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:47.950385 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
W20260925 14:55:47.950531 18945 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:47.945515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:48223 (local address 127.9.141.126:35229)
0925 14:55:47.945608 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:47.945610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:47.945686 (+ 76us) server_negotiation.cc:407] Connection header received
0925 14:55:47.945756 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:47.945757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:47.945769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:47.945810 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:47.946133 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.946617 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.947321 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:47.947529 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:47.947591 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:47.950468 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:48223: BlockingRecv error: recv got EOF from 127.9.141.67:48223 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:55:48 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:48 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:48.688021 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:48.683048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:53749)
0925 14:55:48.683253 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:48.683267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:48.683343 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:48.683474 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:48.683475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:48.683642 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:48.683696 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:48.683697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.684578 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.684580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:48.685130 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:48.685132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.685464 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.685465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:48.685505 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:48.685935 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:48.685939 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:48.687958 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
W20260925 14:55:48.688205 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:48.683150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:53749 (local address 127.9.141.126:35229)
0925 14:55:48.683268 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:48.683270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:48.683325 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:55:48.683390 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:48.683390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:48.683400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:48.683459 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:48.683765 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.684485 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.685200 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.685397 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.685451 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:48.688105 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:53749: BlockingRecv error: recv got EOF from 127.9.141.65:53749 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":49,"threads_started":1}
Sep 25 14:55:48 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:48 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:48.830160 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:48.825522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:40125)
0925 14:55:48.825635 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:48.825648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:48.825718 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:48.825841 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:48.825842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:48.826032 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:48.826076 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:48.826077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.826821 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.826822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:48.827372 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:48.827374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.827762 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.827763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:48.827823 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:48.828310 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:48.828314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:48.830063 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
W20260925 14:55:48.830206 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:48.825597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:40125 (local address 127.9.141.126:35229)
0925 14:55:48.825660 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:48.825662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:48.825699 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:55:48.825772 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:48.825773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:48.825785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:48.825837 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:48.826162 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.826716 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.827468 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.827674 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.827761 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:48.830143 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:40125: BlockingRecv error: recv got EOF from 127.9.141.66:40125 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:55:48 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:48 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:48.955789 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:48.951045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:36209)
0925 14:55:48.951212 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:48.951224 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:48.951288 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:48.951463 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:48.951464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:48.951622 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:48.951669 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:48.951670 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.952298 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.952299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:48.952841 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:48.952843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.953187 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.953188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:48.953225 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:48.953694 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:48.953699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:48.955729 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
W20260925 14:55:48.955935 18949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:48.951152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:36209 (local address 127.9.141.126:35229)
0925 14:55:48.951216 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:48.951218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:48.951272 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:55:48.951378 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:48.951379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:48.951391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:48.951441 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:48.951735 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.952227 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.952920 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:48.953113 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:48.953176 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:48.955847 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:36209: BlockingRecv error: recv got EOF from 127.9.141.67:36209 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:55:49 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:49 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:49.693231 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:49.688573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:41465)
0925 14:55:49.688721 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:49.688736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:49.688818 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:49.689050 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:49.689051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:49.689231 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:49.689280 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:49.689281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.689929 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.689929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:49.690443 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:49.690444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.690778 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.690779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:49.690818 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:49.691260 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:49.691264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:49.693137 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260925 14:55:49.693357 18953 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:49.688609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:41465 (local address 127.9.141.126:35229)
0925 14:55:49.688754 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:49.688756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:49.688812 (+ 56us) server_negotiation.cc:407] Connection header received
0925 14:55:49.688943 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:49.688944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:49.688957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:49.689025 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:49.689351 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.689832 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.690525 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.690715 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.690767 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:49.693273 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:41465: BlockingRecv error: recv got EOF from 127.9.141.65:41465 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
Sep 25 14:55:49 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:49 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:49.836215 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:49.830695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:44897)
0925 14:55:49.831017 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:49.831030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:49.831195 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:49.831387 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:49.831388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:49.831582 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:49.831628 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:49.831630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.832351 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.832352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:49.833086 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:49.833088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.833473 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.833474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:49.833529 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:49.834071 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:49.834076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:49.836107 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":112,"threads_started":1}
W20260925 14:55:49.836297 18953 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:49.830825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:44897 (local address 127.9.141.126:35229)
0925 14:55:49.831014 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:49.831017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:49.831178 (+ 161us) server_negotiation.cc:407] Connection header received
0925 14:55:49.831260 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:49.831261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:49.831276 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:49.831364 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:49.831714 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.832223 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.833183 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.833389 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.833475 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:49.836192 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:44897: BlockingRecv error: recv got EOF from 127.9.141.66:44897 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 25 14:55:49 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:49 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:49.961467 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:49.956311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:37687)
0925 14:55:49.956635 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:49.956646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:49.956708 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:49.956822 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:49.956823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:49.956979 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:49.957059 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:49.957061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.957924 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.957925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:49.958559 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:49.958561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.958878 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.958879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:49.958917 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:49.959523 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:49.959527 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:49.961411 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":268,"threads_started":1}
W20260925 14:55:49.961583 18953 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:49.956425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:37687 (local address 127.9.141.126:35229)
0925 14:55:49.956496 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:49.956498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:49.956714 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:55:49.956747 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:49.956748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:49.956760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:49.956815 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:49.957119 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.957811 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.958627 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:49.958810 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:49.958864 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:49.961509 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:37687: BlockingRecv error: recv got EOF from 127.9.141.67:37687 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:55:50 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:50 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:50.699517 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:50.693718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:55307)
0925 14:55:50.693879 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:50.693891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:50.693974 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:50.694352 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:50.694354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:50.694517 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:50.694568 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:50.694569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.695345 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.695346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:50.695929 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:50.695931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.696299 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.696300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:50.696340 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:50.697254 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:50.697259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:50.699459 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
W20260925 14:55:50.699703 18957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:50.693982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:55307 (local address 127.9.141.126:35229)
0925 14:55:50.694213 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:50.694215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:50.694223 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:50.694245 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:50.694246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:50.694256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:50.694331 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:50.694671 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.695190 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.696026 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.696227 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.696290 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:50.699617 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:55307: BlockingRecv error: recv got EOF from 127.9.141.65:55307 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":121,"threads_started":1}
Sep 25 14:55:50 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:50 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:50.842144 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:50.836751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:34629)
0925 14:55:50.836914 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:50.836925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:50.837042 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:50.837191 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:50.837192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:50.837354 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:50.837401 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:50.837403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.838251 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.838252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:50.838805 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:50.838807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.839161 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.839162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:50.839207 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:50.839763 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:50.839768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:50.842061 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
W20260925 14:55:50.842263 18957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:50.836867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:34629 (local address 127.9.141.126:35229)
0925 14:55:50.836931 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:50.836934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:50.837039 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:55:50.837095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:50.837096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:50.837110 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:50.837173 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:50.837492 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.838162 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.838888 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.839086 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.839144 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:50.842166 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:34629: BlockingRecv error: recv got EOF from 127.9.141.66:34629 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:55:50 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:50 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:50.967109 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:50.961994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:55717)
0925 14:55:50.962159 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:50.962170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:50.962235 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:50.962392 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:50.962393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:50.962648 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:50.962697 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:50.962699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.963418 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.963419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:50.963954 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:50.963956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.964337 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.964339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:50.964450 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:50.965104 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:50.965108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:50.967053 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
W20260925 14:55:50.967276 18957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:50.962048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:55717 (local address 127.9.141.126:35229)
0925 14:55:50.962113 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:50.962115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:50.962237 (+ 122us) server_negotiation.cc:407] Connection header received
0925 14:55:50.962280 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:50.962281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:50.962293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:50.962360 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:50.962771 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.963281 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.964054 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:50.964241 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:50.964304 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:50.967177 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:55717: BlockingRecv error: recv got EOF from 127.9.141.67:55717 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 14:55:51 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:51 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:51.710178 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:51.700040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:55869)
0925 14:55:51.700313 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:51.700328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:51.700480 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:51.700696 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:51.700698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:51.700984 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:51.701087 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:51.701090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.705330 (+ 4240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.705332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:51.706302 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:51.706306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.706798 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.706799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:51.706868 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:51.707418 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:51.707425 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:51.710014 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":88,"threads_started":1}
W20260925 14:55:51.710242 18961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:51.700107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:55869 (local address 127.9.141.126:35229)
0925 14:55:51.700244 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:51.700247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:51.700432 (+ 185us) server_negotiation.cc:407] Connection header received
0925 14:55:51.700600 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:51.700601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:51.700618 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:51.700685 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:51.704442 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.705234 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.706389 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.706722 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.706798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:51.710140 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:55869: BlockingRecv error: recv got EOF from 127.9.141.65:55869 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":46,"threads_started":1}
Sep 25 14:55:51 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:51 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:51.847618 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:51.842721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:51391)
0925 14:55:51.842881 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:51.842901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:51.842958 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:51.843090 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:51.843091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:51.843273 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:51.843323 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:51.843324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.844000 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.844002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:51.844605 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:51.844608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.844992 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.844992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:51.845036 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:51.845565 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:51.845569 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:51.847559 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
W20260925 14:55:51.847745 18961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:51.842808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:51391 (local address 127.9.141.126:35229)
0925 14:55:51.842854 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:51.842856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:51.842948 (+ 92us) server_negotiation.cc:407] Connection header received
0925 14:55:51.843003 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:51.843004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:51.843015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:51.843069 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:51.843386 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.843880 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.844685 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.844885 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.844979 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:51.847663 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:51391: BlockingRecv error: recv got EOF from 127.9.141.66:51391 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 14:55:51 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:51 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:51.973079 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:51.967623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:33695)
0925 14:55:51.967769 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:51.967782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:51.967846 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:51.967979 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:51.967980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:51.968157 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:51.968205 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:51.968207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.968965 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.968966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:51.969562 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:51.969564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.969964 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.969966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:51.970026 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:51.970586 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:51.970592 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:51.972968 (+ 2376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
W20260925 14:55:51.973115 18961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:51.967723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:33695 (local address 127.9.141.126:35229)
0925 14:55:51.967814 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:51.967816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:51.967828 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:55:51.967907 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:51.967908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:51.967921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:51.967970 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:51.968274 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.968883 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.969644 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:51.969862 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:51.969937 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:51.973051 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:33695: BlockingRecv error: recv got EOF from 127.9.141.67:33695 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 14:55:52 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:52 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:52.716171 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:52.710828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:51129)
0925 14:55:52.710978 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:52.710988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:52.711037 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:52.711187 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:52.711188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:52.711358 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:52.711407 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:52.711409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.712118 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.712119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:52.712989 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:52.712993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.713420 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.713421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:52.713470 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:52.713974 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:52.713979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:52.716112 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260925 14:55:52.716339 18965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:52.710916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:51129 (local address 127.9.141.126:35229)
0925 14:55:52.711059 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:52.711061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:52.711070 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:52.711092 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:52.711093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:52.711103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:52.711175 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:52.711480 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.712036 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.713087 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.713324 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.713396 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:52.716227 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:51129: BlockingRecv error: recv got EOF from 127.9.141.65:51129 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
Sep 25 14:55:52 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:52 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:52.857910 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:52.848189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:41793)
0925 14:55:52.848369 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:52.848447 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:52.848523 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:52.848682 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:52.848683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:52.848865 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:52.848911 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:52.848913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.849586 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.849587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:52.850116 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:52.850118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.850532 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.850533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:52.850588 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:52.855859 (+ 5271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:52.855864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:52.857796 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
W20260925 14:55:52.857960 18965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:52.848245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:41793 (local address 127.9.141.126:35229)
0925 14:55:52.848313 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:52.848315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:52.848504 (+ 189us) server_negotiation.cc:407] Connection header received
0925 14:55:52.848590 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:52.848591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:52.848604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:52.848678 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:52.848988 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.849495 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.850233 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.850441 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.850531 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:52.857893 (+ 7362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:41793: BlockingRecv error: recv got EOF from 127.9.141.66:41793 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 14:55:52 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:52 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:52.978502 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:52.973777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:54783)
0925 14:55:52.973946 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:52.973959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:52.974026 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:52.974184 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:52.974185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:52.974371 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:52.974418 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:52.974420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.975080 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.975082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:52.975614 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:52.975615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.975986 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.975987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:52.976045 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:52.976537 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:52.976542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:52.978444 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
W20260925 14:55:52.978576 18965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:52.973909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:54783 (local address 127.9.141.126:35229)
0925 14:55:52.973959 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:52.973962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:52.974008 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:55:52.974082 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:52.974083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:52.974096 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:52.974175 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:52.974503 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.974992 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.975703 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:52.975897 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:52.975985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:52.978527 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:54783: BlockingRecv error: recv got EOF from 127.9.141.67:54783 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 14:55:53 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:53 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:53.721905 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:53.716686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:58123)
0925 14:55:53.716824 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:53.716836 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:53.716898 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:53.717042 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:53.717043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:53.717220 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:53.717270 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:53.717272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.717929 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.717930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:53.718472 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:53.718474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.718938 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.718939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:53.719005 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:53.719572 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:53.719577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:53.721837 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260925 14:55:53.722061 18969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:53.716758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:58123 (local address 127.9.141.126:35229)
0925 14:55:53.716893 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:53.716895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:53.716903 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:53.716952 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:53.716953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:53.716962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:53.717024 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:53.717334 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.717846 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.718552 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.718844 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.718916 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:53.721935 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:58123: BlockingRecv error: recv got EOF from 127.9.141.65:58123 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
Sep 25 14:55:53 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:53 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:53.863229 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:53.858405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:39491)
0925 14:55:53.858580 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:53.858593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:53.858690 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:53.858824 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:53.858825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:53.859016 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:53.859060 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:53.859062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.859780 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.859781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:53.860433 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:53.860436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.860808 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.860809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:53.860865 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:53.861302 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:53.861306 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:53.863162 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
W20260925 14:55:53.863333 18969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:53.858504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:39491 (local address 127.9.141.126:35229)
0925 14:55:53.858584 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:53.858587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:53.858674 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:55:53.858753 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:53.858754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:53.858766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:53.858819 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:53.859165 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.859679 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.860518 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.860722 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.860809 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:53.863262 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:39491: BlockingRecv error: recv got EOF from 127.9.141.66:39491 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 14:55:53 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:53 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:53.984534 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:53.979097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:33309)
0925 14:55:53.979271 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:53.979285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:53.979395 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:53.979653 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:53.979654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:53.979906 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:53.979967 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:53.979969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.980783 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.980784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:53.981454 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:53.981456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.981934 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.981935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:53.981993 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:53.982552 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:53.982558 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:53.984465 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":126,"threads_started":1}
W20260925 14:55:53.984725 18969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:53.979233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:33309 (local address 127.9.141.126:35229)
0925 14:55:53.979282 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:53.979285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:53.979408 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:55:53.979467 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:53.979468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:53.979490 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:53.979596 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:53.980058 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.980653 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.981549 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:53.981843 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:53.981904 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:53.984636 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:33309: BlockingRecv error: recv got EOF from 127.9.141.67:33309 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 14:55:54 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:54 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:54.727375 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:54.722415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:57141)
0925 14:55:54.722601 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:54.722610 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:54.722653 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:54.722876 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:54.722877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:54.723044 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:54.723092 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:54.723093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.723848 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.723848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:54.724451 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:54.724453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.724876 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.724877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:54.724925 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:54.725503 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:54.725508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:54.727311 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
W20260925 14:55:54.727581 18973 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:54.722554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:57141 (local address 127.9.141.126:35229)
0925 14:55:54.722709 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:54.722711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:54.722720 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:54.722752 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:54.722752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:54.722764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:54.722833 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:54.723210 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.723765 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.724553 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.724792 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.724861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:54.727477 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:57141: BlockingRecv error: recv got EOF from 127.9.141.65:57141 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
Sep 25 14:55:54 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:54 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:54.869114 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:54.863761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:49687)
0925 14:55:54.863990 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:54.864004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:54.864091 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:54.864258 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:54.864259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:54.864539 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:54.864600 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:54.864602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.865328 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.865329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:54.865900 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:54.865902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.866392 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.866393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:54.866449 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:54.867061 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:54.867067 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:54.869054 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
W20260925 14:55:54.869251 18973 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:54.863900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:49687 (local address 127.9.141.126:35229)
0925 14:55:54.863970 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:54.863972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:54.864095 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:55:54.864151 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:54.864152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:54.864164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:54.864220 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:54.864682 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.865228 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.865987 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.866259 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.866321 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:54.869164 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:49687: BlockingRecv error: recv got EOF from 127.9.141.66:49687 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:55:54 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:54 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:54.989472 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:54.985005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:56861)
0925 14:55:54.985125 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:54.985135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:54.985190 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:54.985306 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:54.985306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:54.985475 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:54.985522 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:54.985524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.986160 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.986161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:54.986720 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:54.986722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.987051 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.987051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:54.987088 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:54.987632 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:54.987636 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:54.989409 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
W20260925 14:55:54.989643 18973 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:54.985055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:56861 (local address 127.9.141.126:35229)
0925 14:55:54.985126 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:54.985128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:54.985192 (+ 64us) server_negotiation.cc:407] Connection header received
0925 14:55:54.985244 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:54.985244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:54.985256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:54.985291 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:54.985585 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.986082 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.986796 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:54.986988 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:54.987038 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:54.989556 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:56861: BlockingRecv error: recv got EOF from 127.9.141.67:56861 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:55:55 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:55 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:55.732532 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:55.727885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:57063)
0925 14:55:55.728041 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:55.728052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:55.728112 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:55.728282 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:55.728284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:55.728469 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:55.728516 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:55.728518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.729182 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.729183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:55.729710 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:55.729711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.730121 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.730122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:55.730170 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:55.730666 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:55.730671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:55.732466 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
W20260925 14:55:55.732697 18977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:55.727959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:57063 (local address 127.9.141.126:35229)
0925 14:55:55.728108 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:55.728110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:55.728120 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:55:55.728159 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:55.728160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:55.728170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:55.728234 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:55.728608 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.729111 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.729841 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.730037 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.730112 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:55.732593 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:57063: BlockingRecv error: recv got EOF from 127.9.141.65:57063 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
Sep 25 14:55:55 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:55 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:55.874782 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:55.869637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:50197)
0925 14:55:55.869786 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:55.869797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:55.869874 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:55.870039 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:55.870039 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:55.870223 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:55.870271 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:55.870273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.871022 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.871023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:55.871612 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:55.871614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.872004 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.872005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:55.872072 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:55.872617 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:55.872622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:55.874660 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260925 14:55:55.874823 18977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:55.869791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:50197 (local address 127.9.141.126:35229)
0925 14:55:55.869907 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:55.869909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:55.869918 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:55.869946 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:55.869948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:55.869965 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:55.870030 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:55.870410 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.870927 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.871706 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.871915 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.872004 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:55.874775 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:50197: BlockingRecv error: recv got EOF from 127.9.141.66:50197 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 14:55:55 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:55 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:55.994982 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:55.989945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:36253)
0925 14:55:55.990104 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:55.990114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:55.990172 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:55.990306 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:55.990306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:55.990470 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:55.990516 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:55.990517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.991192 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.991192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:55.991841 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:55.991843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.992263 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.992264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:55.992313 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:55.992882 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:55.992888 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:55.994852 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260925 14:55:55.995059 18977 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:55.990020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:36253 (local address 127.9.141.126:35229)
0925 14:55:55.990090 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:55.990091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:55.990176 (+ 85us) server_negotiation.cc:407] Connection header received
0925 14:55:55.990224 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:55.990225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:55.990238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:55.990284 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:55.990581 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.991114 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.991938 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:55.992163 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:55.992228 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:55.995011 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:36253: BlockingRecv error: recv got EOF from 127.9.141.67:36253 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:55:56 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:56 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:56.738302 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:56.733081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:47507)
0925 14:55:56.733239 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:56.733253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:56.733321 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:56.733511 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:56.733512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:56.733796 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:56.733858 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:56.733860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.734528 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.734528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:56.735062 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:56.735063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.735467 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.735468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:56.735522 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:56.736002 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:56.736008 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:56.738194 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
W20260925 14:55:56.738435 18981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:56.733177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:47507 (local address 127.9.141.126:35229)
0925 14:55:56.733367 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:56.733369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:56.733378 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:56.733403 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:56.733404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:56.733416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:56.733486 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:56.733935 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.734426 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.735167 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.735369 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.735435 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:56.738319 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:47507: BlockingRecv error: recv got EOF from 127.9.141.65:47507 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":115,"threads_started":1}
Sep 25 14:55:56 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:56 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:56.880285 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:56.875374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:58221)
0925 14:55:56.875543 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:56.875556 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:56.875627 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:56.875830 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:56.875831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:56.876053 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:56.876111 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:56.876114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.876948 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.876949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:56.877483 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:56.877484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.877873 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.877874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:56.877934 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:56.878445 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:56.878450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:56.880189 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":113,"threads_started":1}
W20260925 14:55:56.880340 18981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:56.875519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:58221 (local address 127.9.141.126:35229)
0925 14:55:56.875569 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:56.875572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:56.875606 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:55:56.875712 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:56.875713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:56.875726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:56.875791 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:56.876188 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.876865 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.877574 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.877785 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.877871 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:56.880269 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:58221: BlockingRecv error: recv got EOF from 127.9.141.66:58221 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 14:55:56 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:56 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.67@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:57.000810 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:56.995743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.67:43623)
0925 14:55:56.995919 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:56.995933 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:56.996011 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:56.996324 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:56.996326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:56.996575 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:56.996622 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:56.996623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.997350 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.997351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:56.997870 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:56.997871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.998309 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.998310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:56.998370 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:56.998842 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:56.998847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:57.000752 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
W20260925 14:55:57.000941 18981 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:56.995818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:43623 (local address 127.9.141.126:35229)
0925 14:55:56.995878 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:56.995880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:56.995991 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:55:56.996090 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:56.996091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:56.996109 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:56.996242 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:56.996727 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.997269 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.997971 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:56.998236 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:56.998298 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:57.000852 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.67:43623: BlockingRecv error: recv got EOF from 127.9.141.67:43623 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 14:55:57 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:57 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.65@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:57.744254 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:57.738954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.65:58003)
0925 14:55:57.739112 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:57.739122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:57.739171 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:57.739367 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:57.739368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:57.739527 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:57.739575 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:57.739577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.740774 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.740775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:57.741410 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:57.741413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.741809 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.741810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:57.741856 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:57.742324 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:57.742329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:57.744199 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260925 14:55:57.744449 18985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:57.739080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:58003 (local address 127.9.141.126:35229)
0925 14:55:57.739214 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:57.739216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:57.739225 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:55:57.739261 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:57.739261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:57.739273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:57.739344 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:57.739668 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.740695 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.741489 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.741731 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.741797 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:57.744338 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.65:58003: BlockingRecv error: recv got EOF from 127.9.141.65:58003 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
Sep 25 14:55:57 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:55:57 dist-test-slave-1sv7 krb5kdc[17927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.141.66@KRBTEST.COM for oryx/127.9.141.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:55:57.889227 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:57.880857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35229 (local address 127.9.141.66:60179)
0925 14:55:57.880983 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:57.880997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:57.881058 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:57.881217 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:57.881217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:57.881484 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:57.881552 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:57.881554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.882328 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.882329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:57.882969 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:57.882972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.883314 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.883315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:57.883359 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:57.887279 (+ 3920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:55:57.887285 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:57.889169 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.141.126:35229: Server oryx/127.9.141.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
W20260925 14:55:57.889370 18985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:57.880949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:60179 (local address 127.9.141.126:35229)
0925 14:55:57.881005 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:57.881007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:57.881044 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:55:57.881104 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:57.881105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:57.881114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:57.881205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:57.881623 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.882240 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.883059 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:57.883247 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:57.883302 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:57.889270 (+ 5968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.141.66:60179: BlockingRecv error: recv got EOF from 127.9.141.66:60179 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:55:57.903517 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18511
I20260925 14:55:57.908027 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18662
I20260925 14:55:57.912505 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18794
I20260925 14:55:57.916769 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18440
2026-09-25T14:55:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17511 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:57Z Disabled control of system clock
I20260925 14:55:57.945276 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35717
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:55:58.034785 18990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.035010 18990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.035032 18990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.036267 18990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:58.036304 18990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.036319 18990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.036334 18990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:58.036347 18990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:58.037819 18990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:35717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.038014 18990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.038326 18990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.043672 18995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.043715 18990 server_base.cc:1034] running on GCE node
W20260925 14:55:58.043665 18998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.043686 18996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.044293 18990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.044610 18990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.045753 18990 hybrid_clock.cc:648] HybridClock initialized: now 1790348158045732 us; error 39 us; skew 500 ppm
I20260925 14:55:58.047009 18990 webserver.cc:460] Webserver started at http://127.9.141.126:44731/ using document root <none> and password file <none>
I20260925 14:55:58.047266 18990 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.047336 18990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.047489 18990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.048646 18990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "04c4758da739487198998a5de47459e7"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "021dde2c936aefd0e2c31167d542e1f6"
server_key_iv: "526cf695fae63fa3571257dc0ce3ceae"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.049005 18990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "04c4758da739487198998a5de47459e7"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "021dde2c936aefd0e2c31167d542e1f6"
server_key_iv: "526cf695fae63fa3571257dc0ce3ceae"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.050323 18990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:58.051023 19004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.051198 18990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:55:58.051288 18990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "04c4758da739487198998a5de47459e7"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "021dde2c936aefd0e2c31167d542e1f6"
server_key_iv: "526cf695fae63fa3571257dc0ce3ceae"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.051362 18990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.057639 18990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.057911 18990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.058058 18990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.062028 18990 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:35717
I20260925 14:55:58.062098 19056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:35717 every 8 connection(s)
I20260925 14:55:58.062376 18990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:58.063302 19057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:58.065560 19057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7: Bootstrap starting.
I20260925 14:55:58.066139 19057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:58.066437 19057 log.cc:824] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:58.067199 19057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7: No bootstrap required, opened a new log
I20260925 14:55:58.068912 19057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4758da739487198998a5de47459e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35717 } }
I20260925 14:55:58.069039 19057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:58.069061 19057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c4758da739487198998a5de47459e7, State: Initialized, Role: FOLLOWER
I20260925 14:55:58.069231 19057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4758da739487198998a5de47459e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35717 } }
I20260925 14:55:58.069332 19057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:58.069387 19057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:58.069435 19057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:58.069690 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 18990
I20260925 14:55:58.069785 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:58.070060 9781 external_mini_cluster.cc:1428] Setting key 2837f406b940c5fac8e93b4dff68cbdc
I20260925 14:55:58.070204 19057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4758da739487198998a5de47459e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35717 } }
I20260925 14:55:58.070333 19057 leader_election.cc:302] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c4758da739487198998a5de47459e7; no voters:
I20260925 14:55:58.070494 19057 leader_election.cc:288] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:58.070523 19060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:58.070642 19060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [term 1 LEADER]: Becoming Leader. State: Replica: 04c4758da739487198998a5de47459e7, State: Running, Role: LEADER
I20260925 14:55:58.070742 19057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:58.070726 19060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4758da739487198998a5de47459e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35717 } }
I20260925 14:55:58.071049 19060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c4758da739487198998a5de47459e7. Latest consensus state: current_term: 1 leader_uuid: "04c4758da739487198998a5de47459e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4758da739487198998a5de47459e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35717 } } }
I20260925 14:55:58.071117 19060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:58.071264 19063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c4758da739487198998a5de47459e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c4758da739487198998a5de47459e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35717 } } }
I20260925 14:55:58.071396 19063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:58.071769 19067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:58.072289 19067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:58.074107 19067 catalog_manager.cc:1347] Generated new cluster ID: b1922afc522a4d01bfbf0a68fffbde47
I20260925 14:55:58.074160 19067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:58.076794 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.070768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.9.141.126:35717)
0925 14:55:58.070994 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.071003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.071413 (+ 410us) server_negotiation.cc:407] Connection header received
0925 14:55:58.071886 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.071893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.071963 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.072104 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.072337 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.073703 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.074698 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.075078 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.075206 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.076105 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.076108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.076108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.076281 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.076312 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.076323 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.076445 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.076525 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.076605 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260925 14:55:58.085486 19067 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:58.085996 19067 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:58.092917 19067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c4758da739487198998a5de47459e7: Generated new TSK 0
I20260925 14:55:58.093107 19067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:58.096966 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35717
--builtin_ntp_servers=127.9.141.84:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:55:58.176644 19081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.176841 19081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.176862 19081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.177999 19081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.178037 19081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.178061 19081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:58.179384 19081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.179613 19081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.179950 19081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.185750 19086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.185770 19087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.186043 19089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.186167 19081 server_base.cc:1034] running on GCE node
I20260925 14:55:58.186367 19081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.186614 19081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.187773 19081 hybrid_clock.cc:648] HybridClock initialized: now 1790348158187761 us; error 28 us; skew 500 ppm
I20260925 14:55:58.189270 19081 webserver.cc:460] Webserver started at http://127.9.141.65:44365/ using document root <none> and password file <none>
I20260925 14:55:58.189540 19081 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.189599 19081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.189721 19081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.191082 19081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "fe06671559904c5ab181ddaa90458b98"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "7dee840da22b3852619afef74e66048e"
server_key_iv: "fc0338b649ad1473fb823f1cade3491b"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.191504 19081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "fe06671559904c5ab181ddaa90458b98"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "7dee840da22b3852619afef74e66048e"
server_key_iv: "fc0338b649ad1473fb823f1cade3491b"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.193331 19081 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:55:58.194250 19095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.194486 19081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:58.194578 19081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "fe06671559904c5ab181ddaa90458b98"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "7dee840da22b3852619afef74e66048e"
server_key_iv: "fc0338b649ad1473fb823f1cade3491b"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.194646 19081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.212195 19081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.212575 19081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.212687 19081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.212891 19081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.213163 19081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.213194 19081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.213215 19081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.213228 19081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.219391 19081 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:35655
I20260925 14:55:58.219434 19208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:35655 every 8 connection(s)
I20260925 14:55:58.219776 19081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:58.221997 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19081
I20260925 14:55:58.222069 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:58.222259 9781 external_mini_cluster.cc:1428] Setting key 57c4ae27880112784bb0d4dd644c2ea4
I20260925 14:55:58.223229 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35717
--builtin_ntp_servers=127.9.141.84:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:55:58.225211 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.220650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:39261 (local address 127.9.141.126:35717)
0925 14:55:58.220722 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.220725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.221001 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:55:58.221453 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.221454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.221466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.221520 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.222174 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.222789 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.223758 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.223972 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.224060 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.224935 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.224937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.224937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.224942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.224956 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.224960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.225004 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.225107 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.225136 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:55:58.225431 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.220647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35717 (local address 127.9.141.65:39261)
0925 14:55:58.220874 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.220907 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.221370 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.221573 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.221576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.221584 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.222094 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.222100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.222881 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.222882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.223664 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.223666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.224070 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.224071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.224203 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.224785 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:58.224786 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.224814 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:58.224835 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.225015 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.225019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.225022 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.225109 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.225208 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260925 14:55:58.225934 19209 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35717
I20260925 14:55:58.226033 19209 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.226233 19209 heartbeater.cc:511] Master 127.9.141.126:35717 requested a full tablet report, sending...
I20260925 14:55:58.226811 19021 ts_manager.cc:194] Registered new tserver with Master: fe06671559904c5ab181ddaa90458b98 (127.9.141.65:35655)
I20260925 14:55:58.227811 19021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.65:39261
W20260925 14:55:58.303957 19212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.304165 19212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.304186 19212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.305294 19212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.305331 19212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.305356 19212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:58.306594 19212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.306834 19212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.307137 19212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.311988 19217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.312098 19218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.312175 19212 server_base.cc:1034] running on GCE node
W20260925 14:55:58.312114 19220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.312861 19212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.313100 19212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.314272 19212 hybrid_clock.cc:648] HybridClock initialized: now 1790348158314257 us; error 35 us; skew 500 ppm
I20260925 14:55:58.315519 19212 webserver.cc:460] Webserver started at http://127.9.141.66:45557/ using document root <none> and password file <none>
I20260925 14:55:58.315719 19212 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.315763 19212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.315919 19212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.317009 19212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "0ea7be652ae648c59d76f5977eba8cca"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "466fdf1e599f8c9547fd19c6a56e8096"
server_key_iv: "ccda4c9bc3b6583b19045fe70b7a285e"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.317310 19212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "0ea7be652ae648c59d76f5977eba8cca"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "466fdf1e599f8c9547fd19c6a56e8096"
server_key_iv: "ccda4c9bc3b6583b19045fe70b7a285e"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.318578 19212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:55:58.319380 19226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.319564 19212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:58.319659 19212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "0ea7be652ae648c59d76f5977eba8cca"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "466fdf1e599f8c9547fd19c6a56e8096"
server_key_iv: "ccda4c9bc3b6583b19045fe70b7a285e"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.319739 19212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.339294 19212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.339629 19212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.339790 19212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.340047 19212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.340406 19212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.340464 19212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.340504 19212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.340549 19212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.346499 19212 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:41675
I20260925 14:55:58.346542 19339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:41675 every 8 connection(s)
I20260925 14:55:58.346951 19212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:58.348344 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19212
I20260925 14:55:58.348474 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:58.348665 9781 external_mini_cluster.cc:1428] Setting key 6c45f53473b5a6bf6dd733ec8f44aabc
I20260925 14:55:58.349658 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35717
--builtin_ntp_servers=127.9.141.84:44225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:55:58.353204 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.347879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:45225 (local address 127.9.141.126:35717)
0925 14:55:58.347929 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.347931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.348105 (+ 174us) server_negotiation.cc:407] Connection header received
0925 14:55:58.348754 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.348755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.348770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.348851 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.349630 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.350415 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.351823 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.352010 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.352072 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.352958 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.352959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.352960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.352964 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.352977 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.352981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.353032 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.353125 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.353147 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:55:58.353485 19342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.347799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35717 (local address 127.9.141.66:45225)
0925 14:55:58.347990 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.348032 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.348637 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.348943 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.348947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.348958 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.349536 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.349541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.350523 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.350524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.351724 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.351727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.352083 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.352084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.352243 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.352807 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:58.352808 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.352846 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:58.352873 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.353044 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.353049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.353052 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.353122 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.353231 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260925 14:55:58.354013 19340 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35717
I20260925 14:55:58.354105 19340 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.354346 19340 heartbeater.cc:511] Master 127.9.141.126:35717 requested a full tablet report, sending...
I20260925 14:55:58.354801 19021 ts_manager.cc:194] Registered new tserver with Master: 0ea7be652ae648c59d76f5977eba8cca (127.9.141.66:41675)
I20260925 14:55:58.355448 19021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.66:45225
W20260925 14:55:58.432052 19343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.432271 19343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.432305 19343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.433478 19343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.433542 19343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.433584 19343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:58.434896 19343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.435137 19343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.435426 19343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.440358 19348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.440376 19351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.440399 19349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.440460 19343 server_base.cc:1034] running on GCE node
I20260925 14:55:58.440904 19343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.441144 19343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.442305 19343 hybrid_clock.cc:648] HybridClock initialized: now 1790348158442292 us; error 40 us; skew 500 ppm
I20260925 14:55:58.443780 19343 webserver.cc:460] Webserver started at http://127.9.141.67:35555/ using document root <none> and password file <none>
I20260925 14:55:58.444032 19343 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.444113 19343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.444249 19343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.445362 19343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "61f54a992ad74d67a7255add02ed6def"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "e4fc75c80801ed2439b89ab5d30145a7"
server_key_iv: "55649c25511744fa7b4789be51db77ab"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.445741 19343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "61f54a992ad74d67a7255add02ed6def"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "e4fc75c80801ed2439b89ab5d30145a7"
server_key_iv: "55649c25511744fa7b4789be51db77ab"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.447084 19343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:55:58.447865 19357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.448083 19343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:58.448177 19343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "61f54a992ad74d67a7255add02ed6def"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "e4fc75c80801ed2439b89ab5d30145a7"
server_key_iv: "55649c25511744fa7b4789be51db77ab"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.448267 19343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.459647 19343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.459935 19343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.460076 19343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.460296 19343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.460675 19343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.460731 19343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.460762 19343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.460790 19343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.466481 19343 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39841
I20260925 14:55:58.466528 19470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39841 every 8 connection(s)
I20260925 14:55:58.466830 19343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:58.472199 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.467564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:47573 (local address 127.9.141.126:35717)
0925 14:55:58.467606 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.467608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.467918 (+ 310us) server_negotiation.cc:407] Connection header received
0925 14:55:58.468492 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.468493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.468506 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.468567 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.469221 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.469722 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.470722 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.470984 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.471063 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.471842 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.471843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.471844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.471849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.471863 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.471867 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.471914 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.472089 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.472122 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:55:58.472425 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.467581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35717 (local address 127.9.141.67:47573)
0925 14:55:58.467789 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.467821 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.468343 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.468635 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.468639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.468648 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.469126 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.469131 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.469816 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.469817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.470631 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:55:58.470632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:55:58.471069 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:55:58.471071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:55:58.471206 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:55:58.471696 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:58.471697 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.471723 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:58.471744 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.471999 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.472006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.472013 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.472092 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.472191 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260925 14:55:58.472939 19471 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35717
I20260925 14:55:58.473013 19471 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.473150 19471 heartbeater.cc:511] Master 127.9.141.126:35717 requested a full tablet report, sending...
I20260925 14:55:58.473657 19021 ts_manager.cc:194] Registered new tserver with Master: 61f54a992ad74d67a7255add02ed6def (127.9.141.67:39841)
I20260925 14:55:58.473985 19021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.67:47573
I20260925 14:55:58.474964 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19343
I20260925 14:55:58.475039 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:58.475207 9781 external_mini_cluster.cc:1428] Setting key ced65fe2222bc70e1392b09ff92b6f8d
I20260925 14:55:58.476122 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:55:58.477948 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19081
W20260925 14:55:58.477973 19064 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:58.477428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.9.141.126:35717)
0925 14:55:58.477514 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.477515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.477523 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:55:58.477594 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.477595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.477604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.477652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.477834 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55142: BlockingRecv error: recv got EOF from 127.0.0.1:55142 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:55:58.482784 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19212
I20260925 14:55:58.487291 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19343
I20260925 14:55:58.491431 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 18990
2026-09-25T14:55:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (572 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:55:58Z Disabled control of system clock
I20260925 14:55:58.518013 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37837
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:55:58.598901 19482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.599112 19482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.599133 19482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.600219 19482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:55:58.600255 19482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.600268 19482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.600281 19482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:55:58.600292 19482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:55:58.601572 19482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.601822 19482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.602085 19482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.606983 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.607018 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.607137 19482 server_base.cc:1034] running on GCE node
W20260925 14:55:58.607018 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.607460 19482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.607722 19482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.608871 19482 hybrid_clock.cc:648] HybridClock initialized: now 1790348158608855 us; error 28 us; skew 500 ppm
I20260925 14:55:58.610242 19482 webserver.cc:460] Webserver started at http://127.9.141.126:38151/ using document root <none> and password file <none>
I20260925 14:55:58.610487 19482 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.610534 19482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.610692 19482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.611749 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "28a8b90535ad46fdad629cec65a9b1e9"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "0ecc23cb706d934ce3d5f3ffda443775"
server_key_iv: "7e93541cdcb4712fbcce8163da9d9fb0"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.612181 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "28a8b90535ad46fdad629cec65a9b1e9"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "0ecc23cb706d934ce3d5f3ffda443775"
server_key_iv: "7e93541cdcb4712fbcce8163da9d9fb0"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.613596 19482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:58.614385 19496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.614603 19482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:58.614697 19482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "28a8b90535ad46fdad629cec65a9b1e9"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "0ecc23cb706d934ce3d5f3ffda443775"
server_key_iv: "7e93541cdcb4712fbcce8163da9d9fb0"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.614756 19482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.618219 19482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.618451 19482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.618551 19482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.622265 19482 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:37837
I20260925 14:55:58.622293 19548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:37837 every 8 connection(s)
I20260925 14:55:58.622573 19482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:55:58.623198 19549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:55:58.625976 19549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9: Bootstrap starting.
I20260925 14:55:58.626551 19549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9: Neither blocks nor log segments found. Creating new log.
I20260925 14:55:58.626847 19549 log.cc:824] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:55:58.627619 19549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9: No bootstrap required, opened a new log
I20260925 14:55:58.629439 19549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a8b90535ad46fdad629cec65a9b1e9" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37837 } }
I20260925 14:55:58.629613 19549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:55:58.629663 19549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a8b90535ad46fdad629cec65a9b1e9, State: Initialized, Role: FOLLOWER
I20260925 14:55:58.629823 19549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a8b90535ad46fdad629cec65a9b1e9" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37837 } }
I20260925 14:55:58.629910 19549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:55:58.629959 19549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:55:58.630020 19549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:55:58.630604 19549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a8b90535ad46fdad629cec65a9b1e9" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37837 } }
I20260925 14:55:58.630730 19549 leader_election.cc:302] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28a8b90535ad46fdad629cec65a9b1e9; no voters:
I20260925 14:55:58.630939 19549 leader_election.cc:288] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:55:58.631039 19552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:55:58.631222 19549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:55:58.631250 19552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [term 1 LEADER]: Becoming Leader. State: Replica: 28a8b90535ad46fdad629cec65a9b1e9, State: Running, Role: LEADER
I20260925 14:55:58.631328 19552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a8b90535ad46fdad629cec65a9b1e9" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37837 } }
I20260925 14:55:58.631737 19554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28a8b90535ad46fdad629cec65a9b1e9. Latest consensus state: current_term: 1 leader_uuid: "28a8b90535ad46fdad629cec65a9b1e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a8b90535ad46fdad629cec65a9b1e9" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37837 } } }
I20260925 14:55:58.631834 19554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:58.632012 19553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28a8b90535ad46fdad629cec65a9b1e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a8b90535ad46fdad629cec65a9b1e9" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37837 } } }
I20260925 14:55:58.632114 19553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:55:58.632416 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19482
I20260925 14:55:58.632501 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:55:58.632699 9781 external_mini_cluster.cc:1428] Setting key 24e609e15a47b966c9ffd9d5f06e1d5f
I20260925 14:55:58.632126 19561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:55:58.633715 19561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:55:58.635493 19561 catalog_manager.cc:1347] Generated new cluster ID: 0ae3a4faa8d440b98a6c1a6e606d1a44
I20260925 14:55:58.635551 19561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:55:58.636078 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.633211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.9.141.126:37837)
0925 14:55:58.633423 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.633434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.634411 (+ 977us) server_negotiation.cc:407] Connection header received
0925 14:55:58.634556 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.634561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.634602 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.634745 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.635492 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.635495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.635495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.635499 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.635548 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.635564 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.635828 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.635857 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.635909 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260925 14:55:58.650015 19561 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:55:58.650252 19561 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:55:58.656998 19561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28a8b90535ad46fdad629cec65a9b1e9: Generated new TSK 0
I20260925 14:55:58.657214 19561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:55:58.659612 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37837
--builtin_ntp_servers=127.9.141.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:55:58.740162 19573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.740439 19573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.740495 19573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.741675 19573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.741748 19573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.741793 19573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:55:58.743141 19573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.743399 19573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.743696 19573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.749531 19578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.749521 19579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.749521 19581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.749864 19573 server_base.cc:1034] running on GCE node
I20260925 14:55:58.750069 19573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.750305 19573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.751480 19573 hybrid_clock.cc:648] HybridClock initialized: now 1790348158751456 us; error 45 us; skew 500 ppm
I20260925 14:55:58.752835 19573 webserver.cc:460] Webserver started at http://127.9.141.65:36559/ using document root <none> and password file <none>
I20260925 14:55:58.753080 19573 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.753151 19573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.753317 19573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.754390 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "3ccbf689f1cd4706a14b6f31182c61d5"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "d5258b8d0224578cf987c3352e07ec1b"
server_key_iv: "1a2bbc3ed434b7a0868e2cae39f64944"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.754812 19573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccbf689f1cd4706a14b6f31182c61d5"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "d5258b8d0224578cf987c3352e07ec1b"
server_key_iv: "1a2bbc3ed434b7a0868e2cae39f64944"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.756132 19573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:55:58.757027 19587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.757217 19573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:55:58.757318 19573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "3ccbf689f1cd4706a14b6f31182c61d5"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "d5258b8d0224578cf987c3352e07ec1b"
server_key_iv: "1a2bbc3ed434b7a0868e2cae39f64944"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.757400 19573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.760630 19573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.760885 19573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.761031 19573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.761248 19573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.761541 19573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.761593 19573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.761631 19573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.761658 19573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.767679 19573 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:42807
I20260925 14:55:58.767747 19700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:42807 every 8 connection(s)
I20260925 14:55:58.768065 19573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:55:58.771210 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.768936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:43837 (local address 127.9.141.126:37837)
0925 14:55:58.768991 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.768993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.769260 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:55:58.769877 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.769878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.769885 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.769927 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.771019 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.771020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.771020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.771024 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.771038 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.771041 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.771087 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.771147 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.771160 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:55:58.771411 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.768915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37837 (local address 127.9.141.65:43837)
0925 14:55:58.769121 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.769163 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.769761 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.770002 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.770006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.770015 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.770890 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:58.770890 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.770916 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:58.770942 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.771084 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.771086 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.771088 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.771140 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.771186 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260925 14:55:58.771926 19701 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37837
I20260925 14:55:58.771996 19701 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.772164 19701 heartbeater.cc:511] Master 127.9.141.126:37837 requested a full tablet report, sending...
I20260925 14:55:58.772727 19513 ts_manager.cc:194] Registered new tserver with Master: 3ccbf689f1cd4706a14b6f31182c61d5 (127.9.141.65:42807)
I20260925 14:55:58.774315 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19573
I20260925 14:55:58.774415 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:55:58.774602 9781 external_mini_cluster.cc:1428] Setting key ff0fa1a7280e7da6d3ade91f042dc631
I20260925 14:55:58.775514 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37837
--builtin_ntp_servers=127.9.141.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:55:58.856065 19704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.856278 19704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.856300 19704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.857525 19704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.857565 19704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.857592 19704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:55:58.858927 19704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.859186 19704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.859513 19704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.864701 19710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.864660 19709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.864665 19712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.864974 19704 server_base.cc:1034] running on GCE node
I20260925 14:55:58.865183 19704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.865386 19704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.866544 19704 hybrid_clock.cc:648] HybridClock initialized: now 1790348158866530 us; error 28 us; skew 500 ppm
I20260925 14:55:58.867772 19704 webserver.cc:460] Webserver started at http://127.9.141.66:33035/ using document root <none> and password file <none>
I20260925 14:55:58.868010 19704 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.868077 19704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.868235 19704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.869414 19704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "b693774eb88d4abd8b1ccffb8fa36897"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "4b21d0f613755775a24502b70a743157"
server_key_iv: "0b6dcf1537371e9d437713c0f6963701"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.869796 19704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "b693774eb88d4abd8b1ccffb8fa36897"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "4b21d0f613755775a24502b70a743157"
server_key_iv: "0b6dcf1537371e9d437713c0f6963701"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.871399 19704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:55:58.872329 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.872592 19704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:55:58.872709 19704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "b693774eb88d4abd8b1ccffb8fa36897"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "4b21d0f613755775a24502b70a743157"
server_key_iv: "0b6dcf1537371e9d437713c0f6963701"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.872799 19704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.876622 19704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.876928 19704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.877087 19704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.877327 19704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.877661 19704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.877720 19704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.877763 19704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.877794 19704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.884315 19704 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:39731
I20260925 14:55:58.884351 19831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:39731 every 8 connection(s)
I20260925 14:55:58.884754 19704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:55:58.887913 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.885484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:52437 (local address 127.9.141.126:37837)
0925 14:55:58.885533 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:58.885535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:58.885839 (+ 304us) server_negotiation.cc:407] Connection header received
0925 14:55:58.886337 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:58.886339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:58.886345 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:58.886400 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:58.887642 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:58.887644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:58.887644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:58.887648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:58.887664 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:58.887668 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:58.887722 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:58.887828 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:55:58.887843 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:55:58.888132 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:58.885469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37837 (local address 127.9.141.66:52437)
0925 14:55:58.885673 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:58.885737 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:58.886220 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:58.886471 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:58.886475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:58.886484 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:58.887483 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:58.887483 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:58.887510 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:58.887539 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:58.887738 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:58.887742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:58.887744 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:55:58.887838 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:55:58.887901 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260925 14:55:58.888695 19832 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37837
I20260925 14:55:58.888772 19832 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:58.888922 19832 heartbeater.cc:511] Master 127.9.141.126:37837 requested a full tablet report, sending...
I20260925 14:55:58.889343 19513 ts_manager.cc:194] Registered new tserver with Master: b693774eb88d4abd8b1ccffb8fa36897 (127.9.141.66:39731)
I20260925 14:55:58.890120 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19704
I20260925 14:55:58.890194 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:55:58.890430 9781 external_mini_cluster.cc:1428] Setting key 610bfadc395f7d5f886f289d205e1b7d
I20260925 14:55:58.891387 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37837
--builtin_ntp_servers=127.9.141.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:55:58.973457 19835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:55:58.973667 19835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:55:58.973690 19835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:55:58.974829 19835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:55:58.974864 19835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:55:58.974890 19835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:55:58.976176 19835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:55:58.976367 19835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:55:58.976692 19835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:55:58.981731 19840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.981796 19841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:55:58.981760 19843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:55:58.981899 19835 server_base.cc:1034] running on GCE node
I20260925 14:55:58.982149 19835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:55:58.982429 19835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:55:58.983625 19835 hybrid_clock.cc:648] HybridClock initialized: now 1790348158983617 us; error 39 us; skew 500 ppm
I20260925 14:55:58.985037 19835 webserver.cc:460] Webserver started at http://127.9.141.67:45199/ using document root <none> and password file <none>
I20260925 14:55:58.985329 19835 fs_manager.cc:359] Metadata directory not provided
I20260925 14:55:58.985405 19835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:55:58.985562 19835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:55:58.986627 19835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "bddb29a292eb406ab1e192684e7e16e3"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "44b792610e9a26a3b95c3a57f93af66a"
server_key_iv: "0a64a47f8ec7906ecfca70ce0630dfd7"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.986996 19835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "bddb29a292eb406ab1e192684e7e16e3"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "44b792610e9a26a3b95c3a57f93af66a"
server_key_iv: "0a64a47f8ec7906ecfca70ce0630dfd7"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.988471 19835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 14:55:58.989270 19849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.989449 19835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:55:58.989549 19835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "bddb29a292eb406ab1e192684e7e16e3"
format_stamp: "Formatted at 2026-09-25 14:55:58 on dist-test-slave-1sv7"
server_key: "44b792610e9a26a3b95c3a57f93af66a"
server_key_iv: "0a64a47f8ec7906ecfca70ce0630dfd7"
server_key_version: "encryptionkey@0"
I20260925 14:55:58.989639 19835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:55:58.993114 19835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:55:58.993394 19835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:55:58.993537 19835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:55:58.993741 19835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:55:58.994041 19835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:55:58.994102 19835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.994140 19835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:55:58.994167 19835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:55:58.999929 19835 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:38751
I20260925 14:55:58.999967 19962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:38751 every 8 connection(s)
I20260925 14:55:59.000319 19835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:55:59.003221 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.001186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:47175 (local address 127.9.141.126:37837)
0925 14:55:59.001276 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.001279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.001413 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:55:59.001867 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.001868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.001873 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.001941 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.003015 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:55:59.003016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:55:59.003016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:55:59.003020 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:55:59.003033 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:55:59.003037 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:55:59.003092 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:55:59.003155 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:55:59.003169 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 14:55:59.003425 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:55:59.001075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37837 (local address 127.9.141.67:47175)
0925 14:55:59.001265 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:55:59.001307 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:55:59.001783 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:55:59.001996 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:55:59.002000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:55:59.002007 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:55:59.002870 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:55:59.002870 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:55:59.002898 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:55:59.002926 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:55:59.003092 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:55:59.003094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:55:59.003097 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:55:59.003147 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:55:59.003194 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260925 14:55:59.003912 19963 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37837
I20260925 14:55:59.004025 19963 heartbeater.cc:464] Registering TS with master...
I20260925 14:55:59.004206 19963 heartbeater.cc:511] Master 127.9.141.126:37837 requested a full tablet report, sending...
I20260925 14:55:59.004632 19513 ts_manager.cc:194] Registered new tserver with Master: bddb29a292eb406ab1e192684e7e16e3 (127.9.141.67:38751)
I20260925 14:55:59.006064 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19835
I20260925 14:55:59.006143 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:55:59.006276 9781 external_mini_cluster.cc:1428] Setting key 6e9db84b24b00c899376107dd310dc40
I20260925 14:55:59.007290 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:55:59.009272 19570 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:55:59.008644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.9.141.126:37837)
0925 14:55:59.008716 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:55:59.008718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:55:59.008918 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:55:59.008944 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:55:59.008945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:55:59.008951 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:55:59.008997 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:55:59.009209 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37636: BlockingRecv error: recv got EOF from 127.0.0.1:37636 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:55:59.009341 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19573
I20260925 14:55:59.014839 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19704
I20260925 14:55:59.019187 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19835
I20260925 14:55:59.023416 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19482
2026-09-25T14:55:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (532 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:55:59 dist-test-slave-1sv7 krb5kdc[19973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:55:59 dist-test-slave-1sv7 krb5kdc[19973](info): set up 2 sockets
Sep 25 14:55:59 dist-test-slave-1sv7 krb5kdc[19973](info): commencing operation
krb5kdc: starting...
W20260925 14:56:00.093981 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:56:00.215914 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35329
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:35329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:00.294553 19986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:00.294787 19986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:00.294823 19986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:00.295943 19986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:00.296013 19986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:00.296036 19986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:00.296065 19986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:00.296097 19986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:00.297601 19986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:35329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:00.297871 19986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:00.298102 19986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:00.302801 19994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:00.302799 19992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:00.302928 19991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:00.302999 19986 server_base.cc:1034] running on GCE node
I20260925 14:56:00.303334 19986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:00.303618 19986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:00.304752 19986 hybrid_clock.cc:648] HybridClock initialized: now 1790348160304742 us; error 33 us; skew 500 ppm
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:00.307193 19986 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:56:00.307658 19986 webserver.cc:460] Webserver started at http://127.9.141.126:34155/ using document root <none> and password file <none>
I20260925 14:56:00.307890 19986 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:00.307934 19986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:00.308076 19986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:00.309142 19986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "4a509b70a4005e39db3476a9d886e111"
server_key_iv: "d44434884c65e3688b8b243fe05b40db"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.309486 19986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "4a509b70a4005e39db3476a9d886e111"
server_key_iv: "d44434884c65e3688b8b243fe05b40db"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.310817 19986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:56:00.311563 20001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.311803 19986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:00.311875 19986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "4a509b70a4005e39db3476a9d886e111"
server_key_iv: "d44434884c65e3688b8b243fe05b40db"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.311964 19986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:00.322795 19986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:00.324832 19986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:00.324990 19986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:00.329097 19986 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:35329
I20260925 14:56:00.329169 20053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:35329 every 8 connection(s)
I20260925 14:56:00.329499 19986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:00.330444 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 19986
I20260925 14:56:00.330557 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:56:00.330754 9781 external_mini_cluster.cc:1428] Setting key 607ab15a8e2a7413f11e5c83f2accb3b
I20260925 14:56:00.330897 20054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:00.334364 20054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54: Bootstrap starting.
I20260925 14:56:00.335081 20054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:00.335386 20054 log.cc:824] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:00.336119 20054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54: No bootstrap required, opened a new log
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:56:00.338380 20054 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35329 } }
I20260925 14:56:00.338554 20054 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:00.338582 20054 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b4c91fbed04d26a2ffd9a2b6065e54, State: Initialized, Role: FOLLOWER
I20260925 14:56:00.338733 20054 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35329 } }
I20260925 14:56:00.338836 20054 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:00.338871 20054 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:00.338920 20054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:00.339525 20054 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35329 } }
I20260925 14:56:00.339634 20054 leader_election.cc:302] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8b4c91fbed04d26a2ffd9a2b6065e54; no voters:
I20260925 14:56:00.339804 20054 leader_election.cc:288] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:00.339840 20059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:00.339974 20059 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [term 1 LEADER]: Becoming Leader. State: Replica: c8b4c91fbed04d26a2ffd9a2b6065e54, State: Running, Role: LEADER
I20260925 14:56:00.340099 20054 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:00.340119 20059 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35329 } }
I20260925 14:56:00.340545 20059 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8b4c91fbed04d26a2ffd9a2b6065e54. Latest consensus state: current_term: 1 leader_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35329 } } }
I20260925 14:56:00.340662 20060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8b4c91fbed04d26a2ffd9a2b6065e54" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35329 } } }
I20260925 14:56:00.340813 20060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:00.341035 20059 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:00.341507 20068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:00.342203 20068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:56:00.342561 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.331332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.9.141.126:35329)
0925 14:56:00.331540 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.331554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.331579 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:56:00.332068 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.332075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.332123 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.332234 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:00.333129 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.334034 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.335058 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.335558 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.335683 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.338496 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:00.338500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:00.338500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:00.338510 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:00.340626 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.341059 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.341062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.341062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.341293 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.341565 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.341566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.341567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.341901 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:00.342048 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.342212 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.342365 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260925 14:56:00.343910 20068 catalog_manager.cc:1347] Generated new cluster ID: 8b6e6719462d416d99db7129ddc7cb81
I20260925 14:56:00.343974 20068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:00.357167 20068 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:00.357717 20068 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:00.373487 20068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54: Generated new TSK 0
I20260925 14:56:00.373713 20068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:56:00.408553 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35329
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:00.492281 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:00.492501 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:00.492525 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:00.493680 20080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:00.493717 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:00.493743 20080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:56:00.495085 20080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:00.495256 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:00.495421 20080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:00.500488 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:00.500520 20080 server_base.cc:1034] running on GCE node
W20260925 14:56:00.500492 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:00.500504 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:00.500960 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:00.501163 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:00.502305 20080 hybrid_clock.cc:648] HybridClock initialized: now 1790348160502282 us; error 44 us; skew 500 ppm
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:00.504856 20080 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:56:00.505326 20080 webserver.cc:460] Webserver started at http://127.9.141.65:41247/ using document root <none> and password file <none>
I20260925 14:56:00.505510 20080 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:00.505551 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:00.505637 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:00.506611 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "b04b4acc4a824dabaadd01c112096057"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "2883121596cb85127dee0841b5a57907"
server_key_iv: "f9f8d0e03c2b7534fd8834d74929be16"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.506907 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "b04b4acc4a824dabaadd01c112096057"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "2883121596cb85127dee0841b5a57907"
server_key_iv: "f9f8d0e03c2b7534fd8834d74929be16"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.508225 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:56:00.509183 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.509449 20080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:56:00.509524 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "b04b4acc4a824dabaadd01c112096057"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "2883121596cb85127dee0841b5a57907"
server_key_iv: "f9f8d0e03c2b7534fd8834d74929be16"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.509574 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:00.534566 20080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:00.537098 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:00.537247 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:00.537539 20080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:00.537869 20080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:00.537925 20080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.537961 20080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:00.537988 20080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.544600 20080 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:42755
I20260925 14:56:00.544655 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:42755 every 8 connection(s)
I20260925 14:56:00.544996 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:56:00.554095 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 20080
I20260925 14:56:00.554217 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:00.554423 9781 external_mini_cluster.cc:1428] Setting key 02a9383fbce1af3857c4226b9f8f532d
I20260925 14:56:00.554543 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.545792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:58487 (local address 127.9.141.126:35329)
0925 14:56:00.545840 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.545842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.546124 (+ 282us) server_negotiation.cc:407] Connection header received
0925 14:56:00.546665 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.546667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.546679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.546736 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:00.547742 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.548250 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.549300 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.549525 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.549614 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.551797 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:00.551799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:00.551800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:00.551807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:00.553132 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.553555 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.553557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.553557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.553616 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.553871 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.553873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.553873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.553989 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:00.554074 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.554361 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.554451 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:56:00.554750 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.545721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35329 (local address 127.9.141.65:58487)
0925 14:56:00.545990 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.546029 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.546557 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.546812 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.546816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.547410 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:00.547622 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.547627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.548346 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.548347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.549202 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.549204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.549625 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.549626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.549756 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.550311 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:00.550319 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:00.551700 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:00.553234 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:00.553241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:00.553243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:00.553459 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:00.553684 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:00.553685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:00.553686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:00.553801 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:00.554063 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:00.554070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:00.554228 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.554339 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.554490 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":148,"threads_started":1}
I20260925 14:56:00.555253 20209 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35329
I20260925 14:56:00.555358 20209 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:00.555564 20209 heartbeater.cc:511] Master 127.9.141.126:35329 requested a full tablet report, sending...
I20260925 14:56:00.556200 20018 ts_manager.cc:194] Registered new tserver with Master: b04b4acc4a824dabaadd01c112096057 (127.9.141.65:42755)
I20260925 14:56:00.556936 20018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:58487
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:56:00.579638 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35329
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:00.658306 20214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:00.658500 20214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:00.658522 20214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:00.659653 20214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:00.659693 20214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:00.659719 20214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:56:00.661163 20214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:00.661396 20214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:00.661634 20214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:00.666621 20222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:00.666594 20220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:00.666605 20219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:00.666662 20214 server_base.cc:1034] running on GCE node
I20260925 14:56:00.666941 20214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:00.667133 20214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:00.668265 20214 hybrid_clock.cc:648] HybridClock initialized: now 1790348160668253 us; error 28 us; skew 500 ppm
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:00.670755 20214 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:56:00.671183 20214 webserver.cc:460] Webserver started at http://127.9.141.66:36113/ using document root <none> and password file <none>
I20260925 14:56:00.671409 20214 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:00.671453 20214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:00.671620 20214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:00.672686 20214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "5f25b14d60314ec59a56e123e76abb1a"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "ab962b4dcbc61d0af724980825854c78"
server_key_iv: "11e5385d8de97ab6462f7f9f2762657b"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.673027 20214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "5f25b14d60314ec59a56e123e76abb1a"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "ab962b4dcbc61d0af724980825854c78"
server_key_iv: "11e5385d8de97ab6462f7f9f2762657b"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.674369 20214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:00.675228 20229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.675483 20214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:56:00.675598 20214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "5f25b14d60314ec59a56e123e76abb1a"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "ab962b4dcbc61d0af724980825854c78"
server_key_iv: "11e5385d8de97ab6462f7f9f2762657b"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.675680 20214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:00.703007 20214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:00.705607 20214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:00.705789 20214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:00.706053 20214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:00.706387 20214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:00.706447 20214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.706496 20214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:00.706524 20214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.712935 20214 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:34965
I20260925 14:56:00.712972 20342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:34965 every 8 connection(s)
I20260925 14:56:00.713343 20214 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:00.714319 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 20214
I20260925 14:56:00.714444 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:00.714640 9781 external_mini_cluster.cc:1428] Setting key 81bc0167e1ec3720dd0eb2220faf6652
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:56:00.724371 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.714213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:43759 (local address 127.9.141.126:35329)
0925 14:56:00.714288 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.714291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.714487 (+ 196us) server_negotiation.cc:407] Connection header received
0925 14:56:00.715168 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.715169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.715180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.715246 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:00.716278 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.716945 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.718332 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.718660 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.718743 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.721144 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:00.721146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:00.721146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:00.721153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:00.722932 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.723376 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.723377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.723378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.723427 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.723693 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.723694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.723694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.723797 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:00.723849 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.724237 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.724307 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:56:00.724638 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.714200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35329 (local address 127.9.141.66:43759)
0925 14:56:00.714356 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.714400 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.715029 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.715347 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.715354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.715985 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:00.716184 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.716190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.717052 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.717053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.718232 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.718235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.718756 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.718757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.718907 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.719525 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:00.719539 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:00.721050 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:00.723019 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:00.723024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:00.723026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:00.723298 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:00.723516 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:00.723517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:00.723517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:00.723626 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:00.723862 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:00.723866 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:00.724102 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.724211 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.724343 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260925 14:56:00.725183 20343 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35329
I20260925 14:56:00.725274 20343 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:00.725427 20343 heartbeater.cc:511] Master 127.9.141.126:35329 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
I20260925 14:56:00.725914 20018 ts_manager.cc:194] Registered new tserver with Master: 5f25b14d60314ec59a56e123e76abb1a (127.9.141.66:34965)
I20260925 14:56:00.726346 20018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:43759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:56:00.742868 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35329
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:00.822369 20348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:00.822578 20348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:00.822603 20348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:00.823776 20348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:00.823817 20348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:00.823843 20348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:56:00.825371 20348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:00.825611 20348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:00.825855 20348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:00.830964 20353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:00.830962 20354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:00.830994 20348 server_base.cc:1034] running on GCE node
W20260925 14:56:00.830960 20356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:00.831454 20348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:00.831655 20348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:00.832785 20348 hybrid_clock.cc:648] HybridClock initialized: now 1790348160832774 us; error 27 us; skew 500 ppm
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:00.835269 20348 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:56:00.835724 20348 webserver.cc:460] Webserver started at http://127.9.141.67:39101/ using document root <none> and password file <none>
I20260925 14:56:00.835979 20348 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:00.836027 20348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:00.836182 20348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:56:00.837306 20348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "3937989ddce14e07a17381f649d274c9"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "239143914a1db8d6d3b1702099927f6c"
server_key_iv: "63a7c2cca7de4b265129d8f6fdc5d16a"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.837668 20348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "3937989ddce14e07a17381f649d274c9"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "239143914a1db8d6d3b1702099927f6c"
server_key_iv: "63a7c2cca7de4b265129d8f6fdc5d16a"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.838953 20348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:56:00.839700 20363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.839895 20348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:56:00.839982 20348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "3937989ddce14e07a17381f649d274c9"
format_stamp: "Formatted at 2026-09-25 14:56:00 on dist-test-slave-1sv7"
server_key: "239143914a1db8d6d3b1702099927f6c"
server_key_iv: "63a7c2cca7de4b265129d8f6fdc5d16a"
server_key_version: "encryptionkey@0"
I20260925 14:56:00.840034 20348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:00.862576 20348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:00.864825 20348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:00.864943 20348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:00.865124 20348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:00.865414 20348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:00.865444 20348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.865466 20348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:00.865479 20348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:00.871690 20348 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:34465
I20260925 14:56:00.871762 20476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:34465 every 8 connection(s)
I20260925 14:56:00.872054 20348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:00.877970 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 20348
I20260925 14:56:00.878067 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:00.878263 9781 external_mini_cluster.cc:1428] Setting key 09bb69bb603792fcf99b5a0ab3b85546
Sep 25 14:56:00 dist-test-slave-1sv7 krb5kdc[19973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348160, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:56:00.881673 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.872853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:35389 (local address 127.9.141.126:35329)
0925 14:56:00.872912 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.872915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.873102 (+ 187us) server_negotiation.cc:407] Connection header received
0925 14:56:00.873627 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.873628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.873641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.873713 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:00.874781 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.875268 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.876250 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.876502 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.876562 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.879106 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:00.879107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:00.879107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:00.879113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:00.880493 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.880857 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.880858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.880859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.880911 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.881126 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.881126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.881127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.881261 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:00.881315 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.881548 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.881609 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:56:00.881863 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.872806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35329 (local address 127.9.141.67:35389)
0925 14:56:00.872974 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.873016 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.873545 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.873794 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.873797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.874492 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:00.874695 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.874699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.875350 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.875351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.876149 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.876151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.876602 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.876603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.876741 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.877242 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:00.877250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:00.879014 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:00.880561 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:00.880567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:00.880569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:00.880785 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:00.880973 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:00.880973 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:00.880973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:00.881068 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:00.881320 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:00.881324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:00.881437 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.881521 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.881638 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260925 14:56:00.882323 20477 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35329
I20260925 14:56:00.882433 20477 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:00.882608 20477 heartbeater.cc:511] Master 127.9.141.126:35329 requested a full tablet report, sending...
I20260925 14:56:00.882964 20018 ts_manager.cc:194] Registered new tserver with Master: 3937989ddce14e07a17381f649d274c9 (127.9.141.67:34465)
I20260925 14:56:00.883365 20018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:35389
I20260925 14:56:00.890275 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:00.897327 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.891546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.9.141.126:35329)
0925 14:56:00.891635 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.891636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.891689 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:56:00.891738 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.891739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.891747 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.891788 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:00.892492 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.892988 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.893746 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.893926 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.893995 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.895188 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:00.895189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:00.895190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:00.895195 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:00.896321 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.896655 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.896656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.896656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.896705 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:00.896868 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:00.896868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:00.896868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:00.896996 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:00.897085 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.897184 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.897238 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:56:00.898815 20018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:00.900503 20018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:00.907549 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.902828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:42755 (local address 127.0.0.1:44590)
0925 14:56:00.903511 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.903521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.903572 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.903881 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.903881 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.903883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:00.903919 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.903920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.905210 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.905212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.906235 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.906238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.907210 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.907211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.907288 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.907303 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.907384 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.907440 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":47,"thread_start_us":98,"threads_started":1}
I20260925 14:56:00.907723 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.902828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34965 (local address 127.0.0.1:48824)
0925 14:56:00.903326 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.903338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.903400 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.903718 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.903719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.903721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:00.903787 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.903788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.905367 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.905367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.906704 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.906706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.907508 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.907508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.907573 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.907576 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.907629 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.907662 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":106,"threads_started":1}
I20260925 14:56:00.907966 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.903103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.9.141.66:34965)
0925 14:56:00.903420 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.903425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.903435 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:56:00.903479 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.903521 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.903542 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.903626 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:00.903873 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.905256 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.906788 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.907363 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.907422 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.907475 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.907841 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.907889 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":194,"threads_started":1}
I20260925 14:56:00.908034 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.902734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:34465 (local address 127.0.0.1:55716)
0925 14:56:00.903069 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.903144 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.903253 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.903595 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.903599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.903601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:00.903674 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.903676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.905130 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.905131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.906460 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.906463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.907761 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.907762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.907830 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.907835 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.907903 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.907960 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":63,"threads_started":1}
I20260925 14:56:00.907570 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.903096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.9.141.65:42755)
0925 14:56:00.903426 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.903431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.903637 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:56:00.903689 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.903691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.903708 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.903782 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:00.904002 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.905087 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.906334 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.907117 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.907204 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.907270 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.907415 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.907464 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260925 14:56:00.908787 20143 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2ed31df8a440688bf77e6c1d08e28 (DEFAULT_TABLE table=test-table [id=a3e911f418af4df6bb62efa98b3a8722]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:00.908788 20277 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2ed31df8a440688bf77e6c1d08e28 (DEFAULT_TABLE table=test-table [id=a3e911f418af4df6bb62efa98b3a8722]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:00.909145 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2ed31df8a440688bf77e6c1d08e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:00.909158 20277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2ed31df8a440688bf77e6c1d08e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:00.911834 20494 tablet_bootstrap.cc:492] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057: Bootstrap starting.
I20260925 14:56:00.911952 20495 tablet_bootstrap.cc:492] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Bootstrap starting.
I20260925 14:56:00.908172 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.902897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.9.141.67:34465)
0925 14:56:00.903116 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.903122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.903227 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:56:00.903389 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.903392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.903411 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.903468 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:00.903762 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.905001 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.906550 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.907315 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.907420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.907478 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.908019 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.908069 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260925 14:56:00.912637 20494 tablet_bootstrap.cc:654] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:00.912648 20495 tablet_bootstrap.cc:654] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:00.912868 20411 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2ed31df8a440688bf77e6c1d08e28 (DEFAULT_TABLE table=test-table [id=a3e911f418af4df6bb62efa98b3a8722]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:00.913242 20494 log.cc:824] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:00.913280 20411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2ed31df8a440688bf77e6c1d08e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:00.913275 20495 log.cc:824] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:00.914095 20494 tablet_bootstrap.cc:492] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057: No bootstrap required, opened a new log
I20260925 14:56:00.914184 20495 tablet_bootstrap.cc:492] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: No bootstrap required, opened a new log
I20260925 14:56:00.914173 20494 ts_tablet_manager.cc:1397] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:56:00.914253 20495 ts_tablet_manager.cc:1397] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:56:00.915901 20495 raft_consensus.cc:348] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.916061 20495 raft_consensus.cc:374] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:00.916105 20495 raft_consensus.cc:729] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f25b14d60314ec59a56e123e76abb1a, State: Initialized, Role: FOLLOWER
I20260925 14:56:00.916232 20495 consensus_queue.cc:260] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.916527 20495 ts_tablet_manager.cc:1428] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:56:00.916538 20494 raft_consensus.cc:348] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.916674 20343 heartbeater.cc:503] Master 127.9.141.126:35329 was elected leader, sending a full tablet report...
I20260925 14:56:00.916702 20494 raft_consensus.cc:374] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:00.916739 20494 raft_consensus.cc:729] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b04b4acc4a824dabaadd01c112096057, State: Initialized, Role: FOLLOWER
I20260925 14:56:00.916863 20494 consensus_queue.cc:260] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.917032 20499 tablet_bootstrap.cc:492] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Bootstrap starting.
I20260925 14:56:00.917129 20494 ts_tablet_manager.cc:1428] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:56:00.917253 20209 heartbeater.cc:503] Master 127.9.141.126:35329 was elected leader, sending a full tablet report...
I20260925 14:56:00.917896 20499 tablet_bootstrap.cc:654] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:00.918390 20499 log.cc:824] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:00.919181 20499 tablet_bootstrap.cc:492] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: No bootstrap required, opened a new log
I20260925 14:56:00.919265 20499 ts_tablet_manager.cc:1397] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:56:00.920631 20499 raft_consensus.cc:348] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.920764 20499 raft_consensus.cc:374] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:00.920809 20499 raft_consensus.cc:729] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3937989ddce14e07a17381f649d274c9, State: Initialized, Role: FOLLOWER
I20260925 14:56:00.920912 20499 consensus_queue.cc:260] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.921155 20499 ts_tablet_manager.cc:1428] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:56:00.921211 20477 heartbeater.cc:503] Master 127.9.141.126:35329 was elected leader, sending a full tablet report...
W20260925 14:56:00.964061 20344 tablet.cc:2369] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:00.979506 20502 raft_consensus.cc:482] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:00.979656 20502 raft_consensus.cc:504] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.980065 20502 leader_election.cc:288] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b04b4acc4a824dabaadd01c112096057 (127.9.141.65:42755), 5f25b14d60314ec59a56e123e76abb1a (127.9.141.66:34965)
I20260925 14:56:00.984061 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.980252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:42755 (local address 127.9.141.67:44541)
0925 14:56:00.980303 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.980316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.980358 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.980555 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.980556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.980558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:00.980633 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.980635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.981614 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.981615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.982980 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.982983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.983774 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.983775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.983872 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.983874 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.983931 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.983979 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":77}
I20260925 14:56:00.984143 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.980324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:44541 (local address 127.9.141.65:42755)
0925 14:56:00.980363 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.980366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.980413 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:56:00.980441 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.980442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.980451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.980505 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:00.980745 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.981518 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.983070 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.983680 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.983756 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.983765 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.984026 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.984072 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:56:00.984149 20493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.980373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:50441 (local address 127.9.141.66:34965)
0925 14:56:00.980472 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.980474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.980586 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:56:00.980632 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.980633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.980639 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.980681 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:00.980839 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.981518 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.983125 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.983680 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.983756 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.983765 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.984026 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.984072 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 14:56:00.984352 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.980280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34965 (local address 127.9.141.67:50441)
0925 14:56:00.980479 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:00.980491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:00.980575 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:00.980705 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:00.980706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:00.980707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:00.980775 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.980776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.981614 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.981615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.983048 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:00.983050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.983774 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.983775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:00.983861 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.983866 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:00.983939 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:56:00.984033 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":89,"thread_start_us":130,"threads_started":1}
I20260925 14:56:00.984531 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2ed31df8a440688bf77e6c1d08e28" candidate_uuid: "3937989ddce14e07a17381f649d274c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04b4acc4a824dabaadd01c112096057" is_pre_election: true
I20260925 14:56:00.984743 20163 raft_consensus.cc:2454] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3937989ddce14e07a17381f649d274c9 in term 0.
I20260925 14:56:00.984701 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2ed31df8a440688bf77e6c1d08e28" candidate_uuid: "3937989ddce14e07a17381f649d274c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f25b14d60314ec59a56e123e76abb1a" is_pre_election: true
I20260925 14:56:00.984833 20297 raft_consensus.cc:2454] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3937989ddce14e07a17381f649d274c9 in term 0.
I20260925 14:56:00.984993 20366 leader_election.cc:302] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3937989ddce14e07a17381f649d274c9, b04b4acc4a824dabaadd01c112096057; no voters:
I20260925 14:56:00.985199 20502 raft_consensus.cc:2781] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:00.985275 20502 raft_consensus.cc:482] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:00.985303 20502 raft_consensus.cc:3037] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:00.986011 20502 raft_consensus.cc:504] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.986173 20502 leader_election.cc:288] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [CANDIDATE]: Term 1 election: Requested vote from peers b04b4acc4a824dabaadd01c112096057 (127.9.141.65:42755), 5f25b14d60314ec59a56e123e76abb1a (127.9.141.66:34965)
I20260925 14:56:00.986311 20297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2ed31df8a440688bf77e6c1d08e28" candidate_uuid: "3937989ddce14e07a17381f649d274c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f25b14d60314ec59a56e123e76abb1a"
I20260925 14:56:00.986328 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2ed31df8a440688bf77e6c1d08e28" candidate_uuid: "3937989ddce14e07a17381f649d274c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b04b4acc4a824dabaadd01c112096057"
I20260925 14:56:00.986398 20163 raft_consensus.cc:3037] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:00.986414 20297 raft_consensus.cc:3037] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:00.987200 20163 raft_consensus.cc:2454] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3937989ddce14e07a17381f649d274c9 in term 1.
I20260925 14:56:00.987313 20297 raft_consensus.cc:2454] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3937989ddce14e07a17381f649d274c9 in term 1.
I20260925 14:56:00.987355 20366 leader_election.cc:302] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3937989ddce14e07a17381f649d274c9, b04b4acc4a824dabaadd01c112096057; no voters:
I20260925 14:56:00.987443 20502 raft_consensus.cc:2781] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:00.987628 20502 raft_consensus.cc:686] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 1 LEADER]: Becoming Leader. State: Replica: 3937989ddce14e07a17381f649d274c9, State: Running, Role: LEADER
I20260925 14:56:00.987725 20502 consensus_queue.cc:237] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } }
I20260925 14:56:00.988641 20018 catalog_manager.cc:5495] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3937989ddce14e07a17381f649d274c9 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "3937989ddce14e07a17381f649d274c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3937989ddce14e07a17381f649d274c9" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 } health_report { overall_health: UNKNOWN } } }
I20260925 14:56:00.999153 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:00.996707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.9.141.67:34465)
0925 14:56:00.996770 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:00.996772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:00.996780 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:00.996853 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:00.996853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:00.996868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:00.996930 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:00.997066 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.997525 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.998429 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:00.998605 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:00.998658 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:00.998817 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:00.998909 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:00.998970 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:00.999015 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:56:00.999031 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:56:01.001024 20297 raft_consensus.cc:1261] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 1 FOLLOWER]: Refusing update from remote peer 3937989ddce14e07a17381f649d274c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:01.001125 20163 raft_consensus.cc:1261] T 9fe2ed31df8a440688bf77e6c1d08e28 P b04b4acc4a824dabaadd01c112096057 [term 1 FOLLOWER]: Refusing update from remote peer 3937989ddce14e07a17381f649d274c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:01.001403 20502 consensus_queue.cc:1035] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f25b14d60314ec59a56e123e76abb1a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:01.001564 20504 consensus_queue.cc:1035] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b04b4acc4a824dabaadd01c112096057" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:01.003754 20507 mvcc.cc:205] Tried to move back new op lower bound from 7333266067457044480 to 7333266067406225408. Current Snapshot: MvccSnapshot[applied={T|T < 7333266067457044480}]
I20260925 14:56:01.004401 20508 mvcc.cc:205] Tried to move back new op lower bound from 7333266067457044480 to 7333266067406225408. Current Snapshot: MvccSnapshot[applied={T|T < 7333266067457044480}]
I20260925 14:56:01.006256 20018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:01.006346 20018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:01.007066 20018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8b4c91fbed04d26a2ffd9a2b6065e54: Sending DeleteTablet for 3 replicas of tablet 9fe2ed31df8a440688bf77e6c1d08e28
I20260925 14:56:01.007510 20143 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe2ed31df8a440688bf77e6c1d08e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:44590
I20260925 14:56:01.007514 20277 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe2ed31df8a440688bf77e6c1d08e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:48824
I20260925 14:56:01.007723 20411 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fe2ed31df8a440688bf77e6c1d08e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55716
I20260925 14:56:01.007748 20517 tablet_replica.cc:330] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: stopping tablet replica
I20260925 14:56:01.007788 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 20080
I20260925 14:56:01.007848 20517 raft_consensus.cc:2229] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:01.009058 20517 raft_consensus.cc:2258] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:01.009511 20517 ts_tablet_manager.cc:1905] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:01.010862 20518 tablet_replica.cc:330] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: stopping tablet replica
I20260925 14:56:01.010959 20518 raft_consensus.cc:2229] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:01.011045 20518 raft_consensus.cc:2258] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:01.011377 20517 ts_tablet_manager.cc:1918] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:01.011449 20517 log.cc:1196] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/9fe2ed31df8a440688bf77e6c1d08e28
I20260925 14:56:01.011482 20518 ts_tablet_manager.cc:1905] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:01.011729 20517 ts_tablet_manager.cc:1939] T 9fe2ed31df8a440688bf77e6c1d08e28 P 5f25b14d60314ec59a56e123e76abb1a: Deleting consensus metadata
I20260925 14:56:01.012194 20003 catalog_manager.cc:4841] TS 5f25b14d60314ec59a56e123e76abb1a (127.9.141.66:34965): tablet 9fe2ed31df8a440688bf77e6c1d08e28 (table test-table [id=a3e911f418af4df6bb62efa98b3a8722]) successfully deleted
I20260925 14:56:01.013388 20518 ts_tablet_manager.cc:1918] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:01.013459 20518 log.cc:1196] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/9fe2ed31df8a440688bf77e6c1d08e28
I20260925 14:56:01.013706 20518 ts_tablet_manager.cc:1939] T 9fe2ed31df8a440688bf77e6c1d08e28 P 3937989ddce14e07a17381f649d274c9: Deleting consensus metadata
I20260925 14:56:01.014261 20005 catalog_manager.cc:4841] TS 3937989ddce14e07a17381f649d274c9 (127.9.141.67:34465): tablet 9fe2ed31df8a440688bf77e6c1d08e28 (table test-table [id=a3e911f418af4df6bb62efa98b3a8722]) successfully deleted
W20260925 14:56:01.015849 20004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:42755 (error 108)
I20260925 14:56:01.016141 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 20214
I20260925 14:56:01.016789 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:01.016239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:42755 (local address 127.0.0.1:44600)
0925 14:56:01.016696 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:01.016743 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:42755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260925 14:56:01.017966 20004 catalog_manager.cc:4584] TS b04b4acc4a824dabaadd01c112096057 (127.9.141.65:42755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fe2ed31df8a440688bf77e6c1d08e28: Network error: Client connection negotiation failed: client connection to 127.9.141.65:42755: connect: Connection refused (error 111)
I20260925 14:56:01.022033 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 20348
I20260925 14:56:01.026778 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 19986
2026-09-25T14:56:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2004 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:01 dist-test-slave-1sv7 krb5kdc[20522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:01 dist-test-slave-1sv7 krb5kdc[20522](info): set up 2 sockets
Sep 25 14:56:01 dist-test-slave-1sv7 krb5kdc[20522](info): commencing operation
krb5kdc: starting...
W20260925 14:56:02.113409 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:56:02 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/rangeradmin_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/rangeradmin_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/rangerlookup_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/rangerlookup_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
I20260925 14:56:02.303049 9781 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5afOW1/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/postgres -l logfile start
2026-09-25 14:56:05.175 UTC [20552] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:56:05.175 UTC [20552] LOG: listening on IPv4 address "127.9.141.84", port 56231
2026-09-25 14:56:05.179 UTC [20552] LOG: listening on Unix socket "/tmp/.s.PGSQL.56231"
2026-09-25 14:56:05.199 UTC [20555] LOG: database system was shut down at 2026-09-25 14:56:03 UTC
2026-09-25 14:56:05.202 UTC [20552] LOG: database system is ready to accept connections
I20260925 14:56:06.173353 9781 mini_postgres.cc:96] Postgres bound to 56231
2026-09-25 14:56:06.179 UTC [20563] FATAL: database "slave" does not exist
127.9.141.84:56231 - accepting connections
I20260925 14:56:06.180321 9781 mini_ranger.cc:161] Starting Ranger...
I20260925 14:56:06.195230 9781 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:56:06.694525 9781 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:56:06.694589 9781 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-admin
2026-09-25 14:56:06,981 [I] DB FLAVOR :POSTGRES
2026-09-25 14:56:06,981 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:56:06,981 [I] Checking connection..
2026-09-25 14:56:06,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:56:07,243 [I] Checking connection passed.
2026-09-25 14:56:07,243 [I] --------- Verifying version history table ---------
2026-09-25 14:56:07,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:56:07,484 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:56:07,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:56:07,738 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:56:07,738 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:56:07,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:56:07.996 UTC [20627] WARNING: there is no transaction in progress
2026-09-25 14:56:08,002 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:56:08,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:56:08,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:56:08,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:56:08,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:56:08,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:56:08,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1sv7.c.gcp-upstream.internal','N') ;"
2026-09-25 14:56:08,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:56:09,209 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:56:09,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:56:09,474 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:56:09,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:56:09,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:56:09,985 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:56:09,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:56:11.014 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.031 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.051 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.069 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.086 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.107 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.171 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.178 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.194 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.204 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.215 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.230 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.241 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.248 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.256 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11.264 UTC [20735] WARNING: there is no transaction in progress
2026-09-25 14:56:11,965 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:56:11,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:56:12,188 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:56:12,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:56:12,427 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:56:12,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:56:12,674 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:56:12,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:56:12,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:56:13,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1sv7.c.gcp-upstream.internal';"
2026-09-25 14:56:13,397 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:56:13,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:56:13,645 [I] DB_PATCHES have already been applied
I20260925 14:56:13.651443 9781 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-admin:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5afOW1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:56:13.651505 9781 mini_ranger.cc:193] Using host: 127.9.141.84
I20260925 14:56:13.654735 9781 mini_ranger.cc:227] Ranger admin URL: http://127.9.141.84:58067
Sep 25, 2026 2:56:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:56:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:56:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:56:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:56:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:56:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:56:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:56:14.799587 9781 mini_ranger.cc:160] Time spent starting Ranger: real 8.619s user 0.003s sys 0.006s
Sep 25, 2026 2:56:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:56:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 2:56:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:56:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:56:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:56:36.094338 9781 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/rangerkms_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/rangerkms_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.141.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.141.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:56:36 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348196, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:56:36.200450 9781 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:56:36.210620 9781 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:56:37.208973 9781 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:56:37.209040 9781 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms
I20260925 14:56:37.209053 9781 mini_ranger_kms.cc:227] Using postgres at 127.9.141.84:56231
2026-09-25 14:56:37,371 [I] DB FLAVOR :POSTGRES
2026-09-25 14:56:37,371 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:56:37,371 [I] Checking connection
2026-09-25 14:56:37,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:56:37,632 [I] connection success
2026-09-25 14:56:37,633 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:56:37,633 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:56:37,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:56:37,884 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:56:37,885 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:56:37,885 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:56:37,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:56:38,159 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:56:38.583837 9781 mini_ranger_kms.cc:310] Created kms service
I20260925 14:56:38.752163 9781 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:56:38.795040 9781 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:56:38 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348196, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:56:39.227356 9781 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:56:39.227478 9781 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms:/tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:56:39.227500 9781 mini_ranger_kms.cc:240] Using host: 127.9.141.84
I20260925 14:56:39.231175 9781 mini_ranger_kms.cc:276] Ranger KMS PID: 20915
I20260925 14:56:39.231302 9781 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.141.84:52203
Sep 25, 2026 2:56:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:56:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:56:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:56:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:56:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:56:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 14:56:41 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348201, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.141.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:41.529270 9781 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.329s user 0.007s sys 0.139s
I20260925 14:56:41.529383 9781 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:56:41.529476 9781 mini_ranger_kms.cc:401] 127.9.141.84:52203/kms/v1/keys
Sep 25 14:56:41 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348201, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.141.84@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
Sep 25 14:56:42 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348201, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.141.84@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
Sep 25 14:56:44 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:56:44.424672 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35181
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:35181
--ranger_config_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:52203/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:44.525194 20966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:44.525442 20966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:44.525482 20966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:44.526721 20966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:56:44.526794 20966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:44.526820 20966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:44.526849 20966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:56:44.526875 20966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:56:44.528538 20966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.141.84:52203/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:35181
--ranger_config_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:44.528811 20966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:44.529058 20966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:44.534548 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:44.534768 20974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:44.534607 20966 server_base.cc:1034] running on GCE node
W20260925 14:56:44.534659 20972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:44.535125 20966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:44.535523 20966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:44.536747 20966 hybrid_clock.cc:648] HybridClock initialized: now 1790348204536717 us; error 47 us; skew 500 ppm
Sep 25 14:56:44 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348204, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:44.539431 20966 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:56:44.539952 20966 webserver.cc:460] Webserver started at http://127.9.141.126:43587/ using document root <none> and password file <none>
I20260925 14:56:44.540215 20966 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:44.540277 20966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:44.540455 20966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:56:44 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348204, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:56:44.586727 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "467583c4fb6040b0805fe249f2f90c41"
format_stamp: "Formatted at 2026-09-25 14:56:44 on dist-test-slave-1sv7"
server_key: "e59a211aaaba176f6b80e4122a3c2ccb"
server_key_iv: "30d3ca1ac16a4f9a5215ec2ea0cbf5b3"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:44.587234 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "467583c4fb6040b0805fe249f2f90c41"
format_stamp: "Formatted at 2026-09-25 14:56:44 on dist-test-slave-1sv7"
server_key: "e59a211aaaba176f6b80e4122a3c2ccb"
server_key_iv: "30d3ca1ac16a4f9a5215ec2ea0cbf5b3"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:44.588841 20966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:56:44.600701 20981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:44.600901 20966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:44.601023 20966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "467583c4fb6040b0805fe249f2f90c41"
format_stamp: "Formatted at 2026-09-25 14:56:44 on dist-test-slave-1sv7"
server_key: "e59a211aaaba176f6b80e4122a3c2ccb"
server_key_iv: "30d3ca1ac16a4f9a5215ec2ea0cbf5b3"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:44.601138 20966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:44.619237 20966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:44.620985 20966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:44.621158 20966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:44.625345 20966 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:35181
I20260925 14:56:44.625377 21033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:35181 every 8 connection(s)
I20260925 14:56:44.625754 20966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:56:44.626405 21034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:44.628728 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41: Bootstrap starting.
I20260925 14:56:44.629251 21034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:44.629482 21034 log.cc:824] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:44.630118 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41: No bootstrap required, opened a new log
I20260925 14:56:44.630355 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 20966
I20260925 14:56:44.630466 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:56:44.630683 9781 external_mini_cluster.cc:1428] Setting key cfb00b3080903d4541aace38001606e1
I20260925 14:56:44.632700 21034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467583c4fb6040b0805fe249f2f90c41" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35181 } }
I20260925 14:56:44.632876 21034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:44.632912 21034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467583c4fb6040b0805fe249f2f90c41, State: Initialized, Role: FOLLOWER
I20260925 14:56:44.633034 21034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467583c4fb6040b0805fe249f2f90c41" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35181 } }
I20260925 14:56:44.633129 21034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:56:44.633164 21034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:56:44.633213 21034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:44.633788 21034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467583c4fb6040b0805fe249f2f90c41" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35181 } }
I20260925 14:56:44.633896 21034 leader_election.cc:302] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 467583c4fb6040b0805fe249f2f90c41; no voters:
I20260925 14:56:44.634042 21034 leader_election.cc:288] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:56:44.634122 21039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:44.634253 21039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [term 1 LEADER]: Becoming Leader. State: Replica: 467583c4fb6040b0805fe249f2f90c41, State: Running, Role: LEADER
I20260925 14:56:44.634279 21034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:56:44.635214 21039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467583c4fb6040b0805fe249f2f90c41" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35181 } }
I20260925 14:56:44.635872 21034 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:56:44.635869 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "467583c4fb6040b0805fe249f2f90c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467583c4fb6040b0805fe249f2f90c41" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35181 } } }
I20260925 14:56:44.635984 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:44.636168 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 467583c4fb6040b0805fe249f2f90c41. Latest consensus state: current_term: 1 leader_uuid: "467583c4fb6040b0805fe249f2f90c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "467583c4fb6040b0805fe249f2f90c41" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35181 } } }
I20260925 14:56:44.636233 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:56:44.638043 21053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:56:44.639246 21053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:56:44 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:56:44.642928 21053 catalog_manager.cc:1347] Generated new cluster ID: 950edd0c7e9747f1813dc028004da9dc
I20260925 14:56:44.643016 21053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:56:44.646129 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:44.631217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.9.141.126:35181)
0925 14:56:44.631452 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:44.631463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:44.631498 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:56:44.632670 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:44.632680 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:44.632744 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:44.632901 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:44.633721 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:44.634997 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:44.636212 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:44.638954 (+ 2742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:44.639176 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:44.642133 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:44.642140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:44.642140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:44.642150 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:44.644318 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:44.644910 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:44.644913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:44.644914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:44.645017 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:44.645233 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:44.645234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:44.645234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:44.645512 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:44.645675 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:44.645782 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:56:44.645914 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
I20260925 14:56:44.669922 21053 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:56:44.670791 21053 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:56:44.681324 21053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41: Generated new TSK 0
I20260925 14:56:44.681527 21053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:56:45.328002 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:52203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35181
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:45.444931 21079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:45.445200 21079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:45.445237 21079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:45.446910 21079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:45.446964 21079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:45.447005 21079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:56:45.449115 21079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.141.84:52203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:45.449493 21079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:45.449744 21079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:45.455986 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:45.456105 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:45.456431 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:45.456609 21079 server_base.cc:1034] running on GCE node
I20260925 14:56:45.456941 21079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:45.457206 21079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:45.458375 21079 hybrid_clock.cc:648] HybridClock initialized: now 1790348205458352 us; error 36 us; skew 500 ppm
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:45.461750 21079 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:56:45.462229 21079 webserver.cc:460] Webserver started at http://127.9.141.65:45389/ using document root <none> and password file <none>
I20260925 14:56:45.462460 21079 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:45.462507 21079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:45.462625 21079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:56:45.477572 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "2aaaeafd77b84626a7564754bdec6e27"
format_stamp: "Formatted at 2026-09-25 14:56:45 on dist-test-slave-1sv7"
server_key: "e797e8d2a6e1ba53776ebf8bbf963173"
server_key_iv: "8641874559ba9e2955ab851d1e9cb790"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:45.478087 21079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "2aaaeafd77b84626a7564754bdec6e27"
format_stamp: "Formatted at 2026-09-25 14:56:45 on dist-test-slave-1sv7"
server_key: "e797e8d2a6e1ba53776ebf8bbf963173"
server_key_iv: "8641874559ba9e2955ab851d1e9cb790"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:45.481737 21079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 14:56:45.503724 21096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:45.503988 21079 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 14:56:45.504091 21079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "2aaaeafd77b84626a7564754bdec6e27"
format_stamp: "Formatted at 2026-09-25 14:56:45 on dist-test-slave-1sv7"
server_key: "e797e8d2a6e1ba53776ebf8bbf963173"
server_key_iv: "8641874559ba9e2955ab851d1e9cb790"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:45.504197 21079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:45.548611 21079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:45.554678 21079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:45.554848 21079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:45.555073 21079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:45.555449 21079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:45.555500 21079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:45.555536 21079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:45.555558 21079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:45.561789 21079 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:39333
I20260925 14:56:45.562193 21079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:56:45.569378 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 21079
I20260925 14:56:45.569538 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:56:45.569766 9781 external_mini_cluster.cc:1428] Setting key cdbdc2f88ccb90795d4495a195bc1b59
I20260925 14:56:45.576676 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:39333 every 8 connection(s)
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
I20260925 14:56:45.591735 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.577450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35181 (local address 127.9.141.65:47153)
0925 14:56:45.580438 (+ 2988us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:45.580496 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:45.581259 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:45.581727 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:45.581732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:45.582007 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:45.582297 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:45.582304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.583336 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.583338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:45.584832 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:45.584834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.585419 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.585420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:45.585578 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:45.586220 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:45.586231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:45.587908 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:45.590072 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:45.590079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:45.590082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:45.590368 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:45.590605 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:45.590605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:45.590606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:45.590750 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:45.591052 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:45.591058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:45.591218 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 14:56:45.591329 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:56:45.591482 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2902,"thread_start_us":110,"threads_started":1}
I20260925 14:56:45.592092 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.577781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:47153 (local address 127.9.141.126:35181)
0925 14:56:45.581422 (+ 3641us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:45.581424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:45.581436 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:56:45.581464 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:45.581465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:45.581479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:45.581563 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:45.582420 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.583218 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.584945 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.585278 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.585348 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:45.588029 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:45.588031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:45.588031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:45.588040 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:45.589975 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:45.590455 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:45.590457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:45.590457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:45.590528 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:45.590826 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:45.590827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:45.590827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:45.590966 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:45.591025 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:45.591897 (+ 872us) server_negotiation.cc:299] Negotiation successful
0925 14:56:45.591984 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3607,"thread_start_us":88,"threads_started":1}
I20260925 14:56:45.592537 21210 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35181
I20260925 14:56:45.592630 21210 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:45.592828 21210 heartbeater.cc:511] Master 127.9.141.126:35181 requested a full tablet report, sending...
I20260925 14:56:45.593394 20998 ts_manager.cc:194] Registered new tserver with Master: 2aaaeafd77b84626a7564754bdec6e27 (127.9.141.65:39333)
I20260925 14:56:45.594249 20998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:47153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:56:45.636046 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:52203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35181
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:56:45.757401 21218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:45.757761 21218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:45.757838 21218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:45.759578 21218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:45.759690 21218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:45.759774 21218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:56:45.761909 21218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.141.84:52203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:45.762344 21218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:45.762634 21218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:45.769817 21224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:45.769966 21226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:45.770543 21223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:45.771029 21218 server_base.cc:1034] running on GCE node
I20260925 14:56:45.771232 21218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:45.771468 21218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:45.772615 21218 hybrid_clock.cc:648] HybridClock initialized: now 1790348205772597 us; error 35 us; skew 500 ppm
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:45.775889 21218 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:56:45.776432 21218 webserver.cc:460] Webserver started at http://127.9.141.66:35921/ using document root <none> and password file <none>
I20260925 14:56:45.776676 21218 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:45.776723 21218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:45.776840 21218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:56:45.793708 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "f6bd2d980f49476cb844d04f3a302d16"
format_stamp: "Formatted at 2026-09-25 14:56:45 on dist-test-slave-1sv7"
server_key: "c1bf26c89dca32988eb11fb61f52359f"
server_key_iv: "20b691c61851e8567bfecba4e7cb4a67"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:45.796255 21218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "f6bd2d980f49476cb844d04f3a302d16"
format_stamp: "Formatted at 2026-09-25 14:56:45 on dist-test-slave-1sv7"
server_key: "c1bf26c89dca32988eb11fb61f52359f"
server_key_iv: "20b691c61851e8567bfecba4e7cb4a67"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:45.799467 21218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:56:45.815148 21234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:45.815403 21218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:45.815519 21218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "f6bd2d980f49476cb844d04f3a302d16"
format_stamp: "Formatted at 2026-09-25 14:56:45 on dist-test-slave-1sv7"
server_key: "c1bf26c89dca32988eb11fb61f52359f"
server_key_iv: "20b691c61851e8567bfecba4e7cb4a67"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:45.815623 21218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:45.827713 21218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:45.829929 21218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:45.830096 21218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:45.830363 21218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:45.830753 21218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:45.830808 21218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:45.830868 21218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:45.830895 21218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:45.837993 21218 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:45797
I20260925 14:56:45.838485 21218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:56:45.840137 21347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:45797 every 8 connection(s)
Sep 25 14:56:45 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:56:45.845935 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 21218
I20260925 14:56:45.846087 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:56:45.846581 9781 external_mini_cluster.cc:1428] Setting key eb950ce2b7e018b2a49b359c35781fb5
I20260925 14:56:45.849371 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.839640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:41539 (local address 127.9.141.126:35181)
0925 14:56:45.839696 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:45.839699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:45.839922 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:56:45.840688 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:45.840689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:45.840712 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:45.840764 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:45.841338 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.841865 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.842882 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.843116 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.843197 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:45.845316 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:45.845318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:45.845318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:45.845326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:45.847177 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:45.848129 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:45.848131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:45.848131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:45.848187 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:45.848510 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:45.848511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:45.848512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:45.848656 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:45.848699 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:45.849137 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 14:56:45.849249 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:56:45.859264 21350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:45.839329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35181 (local address 127.9.141.66:41539)
0925 14:56:45.839778 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:45.839832 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:45.840561 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:45.840835 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:45.840840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:45.841035 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:45.841235 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:45.841240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.841968 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.841969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:45.842786 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:45.842788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:45.843203 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:45.843211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:45.843348 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:45.843913 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:45.843922 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:45.845218 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:45.847704 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:45.847710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:45.847713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:45.848037 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:45.848274 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:45.848275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:45.848275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:45.848434 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:45.848746 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:45.848752 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:45.848959 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 14:56:45.858844 (+ 9885us) client_negotiation.cc:240] Negotiation successful
0925 14:56:45.859026 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":83,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
I20260925 14:56:45.859985 21348 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35181
I20260925 14:56:45.860088 21348 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:45.860321 21348 heartbeater.cc:511] Master 127.9.141.126:35181 requested a full tablet report, sending...
I20260925 14:56:45.860783 20998 ts_manager.cc:194] Registered new tserver with Master: f6bd2d980f49476cb844d04f3a302d16 (127.9.141.66:45797)
I20260925 14:56:45.861312 20998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:41539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:56:45.886418 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:52203/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35181
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:56:46.026115 21353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:56:46.027608 21353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:56:46.027732 21353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:56:46.030256 21353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:56:46.030360 21353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:56:46.030437 21353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:56:46.033105 21353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.141.84:52203/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:35181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:56:46.033540 21353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:56:46.033927 21353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:56:46.041118 21359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:46.041522 21361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:56:46.041610 21358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:56:46.042593 21353 server_base.cc:1034] running on GCE node
I20260925 14:56:46.042907 21353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:56:46.043174 21353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:56:46.044355 21353 hybrid_clock.cc:648] HybridClock initialized: now 1790348206044334 us; error 35 us; skew 500 ppm
Sep 25 14:56:46 dist-test-slave-1sv7 krb5kdc[20522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:56:46.048768 21353 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:56:46.049345 21353 webserver.cc:460] Webserver started at http://127.9.141.67:32939/ using document root <none> and password file <none>
I20260925 14:56:46.049616 21353 fs_manager.cc:359] Metadata directory not provided
I20260925 14:56:46.049675 21353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:56:46.049810 21353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:56:46 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:56:46.070143 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "cf6bb71065f24ffc873651c0bd4d1870"
format_stamp: "Formatted at 2026-09-25 14:56:46 on dist-test-slave-1sv7"
server_key: "a13827512371ef02e85c04f61eba9216"
server_key_iv: "77ed84859b6357babe52f943bad17880"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:46.070715 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "cf6bb71065f24ffc873651c0bd4d1870"
format_stamp: "Formatted at 2026-09-25 14:56:46 on dist-test-slave-1sv7"
server_key: "a13827512371ef02e85c04f61eba9216"
server_key_iv: "77ed84859b6357babe52f943bad17880"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:46.074098 21353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:56:46.090205 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:46.090487 21353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:56:46.090588 21353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "cf6bb71065f24ffc873651c0bd4d1870"
format_stamp: "Formatted at 2026-09-25 14:56:46 on dist-test-slave-1sv7"
server_key: "a13827512371ef02e85c04f61eba9216"
server_key_iv: "77ed84859b6357babe52f943bad17880"
server_key_version: "kuduclusterkey@0"
I20260925 14:56:46.090687 21353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:56:46.117686 21353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:56:46.120155 21353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:56:46.120364 21353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:56:46.120743 21353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:56:46.121155 21353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:56:46.121255 21353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:46.121336 21353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:56:46.121395 21353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:56:46.128182 21353 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39255
I20260925 14:56:46.128791 21353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:56:46.130703 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 21353
I20260925 14:56:46.130807 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:56:46.131016 9781 external_mini_cluster.cc:1428] Setting key 8b120d7b095bc528c2762edc3490b83c
I20260925 14:56:46.133222 21481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39255 every 8 connection(s)
Sep 25 14:56:46 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348206, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:56:46.145596 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.133884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:39419 (local address 127.9.141.126:35181)
0925 14:56:46.133948 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.133950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.134297 (+ 347us) server_negotiation.cc:407] Connection header received
0925 14:56:46.134989 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.134990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.135001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.135033 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:46.135994 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.136780 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.138359 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.138639 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.138688 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.141418 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:46.141419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:46.141420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:46.141428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:46.143929 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.144471 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.144473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.144473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.144532 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.144827 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.144828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.144828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.144963 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:46.145008 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.145408 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.145498 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:56:46.146471 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.133910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:35181 (local address 127.9.141.67:39419)
0925 14:56:46.134082 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.134204 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:46.134862 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:46.135341 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:46.135345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:46.135599 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:56:46.135881 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.135888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.136898 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.136899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.138256 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.138258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.138785 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.138786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.138925 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.139569 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:56:46.139580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:56:46.141295 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:56:46.144026 (+ 2731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:46.144034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:46.144037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:46.144357 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:46.144606 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:56:46.144607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:56:46.144607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:56:46.144744 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:56:46.145058 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:56:46.145063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:56:46.145229 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 14:56:46.145963 (+ 734us) client_negotiation.cc:240] Negotiation successful
0925 14:56:46.146194 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 14:56:46.147199 21482 heartbeater.cc:347] Connected to a master server at 127.9.141.126:35181
I20260925 14:56:46.147306 21482 heartbeater.cc:464] Registering TS with master...
I20260925 14:56:46.147502 21482 heartbeater.cc:511] Master 127.9.141.126:35181 requested a full tablet report, sending...
I20260925 14:56:46.148080 20998 ts_manager.cc:194] Registered new tserver with Master: cf6bb71065f24ffc873651c0bd4d1870 (127.9.141.67:39255)
I20260925 14:56:46.148635 20998 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:39419
I20260925 14:56:46.153930 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:56:46.163880 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.155727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.9.141.126:35181)
0925 14:56:46.155769 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.155771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.156046 (+ 275us) server_negotiation.cc:407] Connection header received
0925 14:56:46.156152 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.156153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.156164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.156196 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:56:46.157018 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.157767 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.158848 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.159136 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.159186 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.160845 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:56:46.160847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:56:46.160847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:56:46.160855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:56:46.162606 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.163013 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.163015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.163015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.163074 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:56:46.163276 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:56:46.163277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:56:46.163277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:56:46.163412 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:56:46.163486 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.163706 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.163783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:56:46.166145 20998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:56:46.168510 20998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:56:46.179886 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.173609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:45797 (local address 127.0.0.1:39618)
0925 14:56:46.174791 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.174799 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:46.174835 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:46.175198 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:46.175202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:46.175204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:46.175274 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.175276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.176919 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.176920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.178522 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.178524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.179582 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.179583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.179667 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.179681 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:56:46.179753 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:56:46.179802 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":63,"threads_started":1}
I20260925 14:56:46.180490 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.173947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.9.141.66:45797)
0925 14:56:46.174947 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.174952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.174963 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:56:46.175003 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.175005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.175021 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.175099 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:46.175377 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.176823 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.178618 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.179376 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.179462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.179544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.180349 (+ 805us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.180409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":79,"threads_started":1}
I20260925 14:56:46.181646 21282 tablet_service.cc:1467] Processing CreateTablet for tablet 28eb0698b5c84c198cfa7abeb567d92b (DEFAULT_TABLE table=test-table [id=a0edd965856c41b5a5e517ea8362cc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:46.182013 21282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eb0698b5c84c198cfa7abeb567d92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:46.187288 21501 tablet_bootstrap.cc:492] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Bootstrap starting.
I20260925 14:56:46.188254 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.173421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39255 (local address 127.0.0.1:53754)
0925 14:56:46.174664 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.174673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:46.174706 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:46.180703 (+ 5997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:46.180704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:46.180707 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:46.180745 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.180746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.185940 (+ 5194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.185941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.187031 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.187034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.187982 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.187983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.188051 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.188056 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:46.188119 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:56:46.188163 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"thread_start_us":75,"threads_started":1}
I20260925 14:56:46.189841 21501 tablet_bootstrap.cc:654] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:46.190217 21501 log.cc:824] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:46.190418 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.173826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.9.141.67:39255)
0925 14:56:46.180472 (+ 6646us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.180477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.180490 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:56:46.180528 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.180530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.180546 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.180629 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:46.180898 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.181933 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.187141 (+ 5208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.187901 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.187982 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.188054 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.189387 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.189435 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6605,"thread_start_us":83,"threads_started":1}
I20260925 14:56:46.191084 21501 tablet_bootstrap.cc:492] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: No bootstrap required, opened a new log
I20260925 14:56:46.191159 21501 ts_tablet_manager.cc:1397] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:56:46.191159 21416 tablet_service.cc:1467] Processing CreateTablet for tablet 28eb0698b5c84c198cfa7abeb567d92b (DEFAULT_TABLE table=test-table [id=a0edd965856c41b5a5e517ea8362cc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:46.191468 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eb0698b5c84c198cfa7abeb567d92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:46.193272 21501 raft_consensus.cc:348] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.193446 21501 raft_consensus.cc:374] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:46.193480 21501 raft_consensus.cc:729] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bd2d980f49476cb844d04f3a302d16, State: Initialized, Role: FOLLOWER
I20260925 14:56:46.193603 21501 consensus_queue.cc:260] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.193872 21501 ts_tablet_manager.cc:1428] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:56:46.194592 21348 heartbeater.cc:503] Master 127.9.141.126:35181 was elected leader, sending a full tablet report...
I20260925 14:56:46.194877 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.173189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39333 (local address 127.0.0.1:45552)
0925 14:56:46.174443 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.174486 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:46.174554 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:46.182053 (+ 7499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:46.182054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:46.182057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:46.182103 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.182104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.188119 (+ 6015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.188120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.189303 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.189306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.194619 (+ 5313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.194621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.194722 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.194728 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:56:46.194759 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:56:46.194801 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":74,"threads_started":1}
I20260925 14:56:46.195429 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.173284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.9.141.65:39333)
0925 14:56:46.180752 (+ 7468us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.180756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.180764 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:56:46.180795 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.180797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.180811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.182174 (+ 1363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:46.182237 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.183255 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.189607 (+ 6352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.190340 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.195190 (+ 4850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.195272 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.195309 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.195356 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7433,"thread_start_us":49,"threads_started":1}
I20260925 14:56:46.196204 21144 tablet_service.cc:1467] Processing CreateTablet for tablet 28eb0698b5c84c198cfa7abeb567d92b (DEFAULT_TABLE table=test-table [id=a0edd965856c41b5a5e517ea8362cc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:56:46.196524 21144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28eb0698b5c84c198cfa7abeb567d92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:56:46.199944 21504 tablet_bootstrap.cc:492] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Bootstrap starting.
I20260925 14:56:46.200507 21505 tablet_bootstrap.cc:492] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: Bootstrap starting.
I20260925 14:56:46.201212 21505 tablet_bootstrap.cc:654] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:46.201526 21505 log.cc:824] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:46.202353 21504 tablet_bootstrap.cc:654] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Neither blocks nor log segments found. Creating new log.
I20260925 14:56:46.202683 21504 log.cc:824] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Log is configured to *not* fsync() on all Append() calls
I20260925 14:56:46.203552 21505 tablet_bootstrap.cc:492] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: No bootstrap required, opened a new log
I20260925 14:56:46.203657 21505 ts_tablet_manager.cc:1397] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:56:46.206749 21504 tablet_bootstrap.cc:492] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: No bootstrap required, opened a new log
I20260925 14:56:46.206890 21504 ts_tablet_manager.cc:1397] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 14:56:46.206915 21505 raft_consensus.cc:348] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.207129 21505 raft_consensus.cc:374] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:46.207207 21505 raft_consensus.cc:729] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aaaeafd77b84626a7564754bdec6e27, State: Initialized, Role: FOLLOWER
I20260925 14:56:46.207408 21505 consensus_queue.cc:260] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.207731 21505 ts_tablet_manager.cc:1428] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:56:46.207887 21210 heartbeater.cc:503] Master 127.9.141.126:35181 was elected leader, sending a full tablet report...
I20260925 14:56:46.209774 21504 raft_consensus.cc:348] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.210001 21504 raft_consensus.cc:374] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:56:46.210147 21504 raft_consensus.cc:729] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6bb71065f24ffc873651c0bd4d1870, State: Initialized, Role: FOLLOWER
I20260925 14:56:46.210359 21504 consensus_queue.cc:260] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.210690 21504 ts_tablet_manager.cc:1428] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:56:46.210843 21482 heartbeater.cc:503] Master 127.9.141.126:35181 was elected leader, sending a full tablet report...
Sep 25 14:56:46 dist-test-slave-1sv7 krb5kdc[20522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348205, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
W20260925 14:56:46.327713 21211 tablet.cc:2369] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:46.346722 21349 tablet.cc:2369] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:56:46.383939 21483 tablet.cc:2369] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:56:46.448096 21509 raft_consensus.cc:482] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:56:46.448251 21509 raft_consensus.cc:504] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.448699 21509 leader_election.cc:288] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aaaeafd77b84626a7564754bdec6e27 (127.9.141.65:39333), f6bd2d980f49476cb844d04f3a302d16 (127.9.141.66:45797)
I20260925 14:56:46.452615 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.448874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39333 (local address 127.9.141.67:37473)
0925 14:56:46.448974 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.448990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:46.449034 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:46.449346 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:46.449347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:46.449349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:46.449415 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.449416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.450327 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.450328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.451537 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.451539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.452331 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.452332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.452437 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.452441 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:56:46.452489 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:56:46.452528 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":7}
I20260925 14:56:46.452998 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.449122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:37473 (local address 127.9.141.65:39333)
0925 14:56:46.449184 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.449186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.449200 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:56:46.449231 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.449232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.449243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.449337 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:46.449486 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.450225 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.451653 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.452223 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.452292 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.452300 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.452898 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.452943 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:56:46.453540 21164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eb0698b5c84c198cfa7abeb567d92b" candidate_uuid: "cf6bb71065f24ffc873651c0bd4d1870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aaaeafd77b84626a7564754bdec6e27" is_pre_election: true
I20260925 14:56:46.453704 21164 raft_consensus.cc:2454] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6bb71065f24ffc873651c0bd4d1870 in term 0.
I20260925 14:56:46.453698 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.449151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:43799 (local address 127.9.141.66:45797)
0925 14:56:46.449195 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.449197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.449271 (+ 74us) server_negotiation.cc:407] Connection header received
0925 14:56:46.449530 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.449531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.449547 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.449577 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:56:46.449781 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.450445 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.451682 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.452231 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.452275 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.452286 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.453582 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.453630 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:56:46.453924 21372 leader_election.cc:302] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aaaeafd77b84626a7564754bdec6e27, cf6bb71065f24ffc873651c0bd4d1870; no voters:
I20260925 14:56:46.454048 21509 raft_consensus.cc:2781] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:56:46.454099 21509 raft_consensus.cc:482] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:56:46.454128 21509 raft_consensus.cc:3037] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:46.455029 21509 raft_consensus.cc:504] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.455166 21509 leader_election.cc:288] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [CANDIDATE]: Term 1 election: Requested vote from peers 2aaaeafd77b84626a7564754bdec6e27 (127.9.141.65:39333), f6bd2d980f49476cb844d04f3a302d16 (127.9.141.66:45797)
I20260925 14:56:46.455374 21164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eb0698b5c84c198cfa7abeb567d92b" candidate_uuid: "cf6bb71065f24ffc873651c0bd4d1870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aaaeafd77b84626a7564754bdec6e27"
I20260925 14:56:46.455426 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.448925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:45797 (local address 127.9.141.67:43799)
0925 14:56:46.449197 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.449206 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:56:46.449283 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:56:46.449638 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:56:46.449639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:56:46.449641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:56:46.449713 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.449714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.450537 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.450538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.451592 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:56:46.451594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.452339 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.452339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:56:46.453484 (+ 1145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.453489 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:56:46.453811 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 14:56:46.455327 (+ 1516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260925 14:56:46.455457 21164 raft_consensus.cc:3037] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:46.455744 21302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eb0698b5c84c198cfa7abeb567d92b" candidate_uuid: "cf6bb71065f24ffc873651c0bd4d1870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bd2d980f49476cb844d04f3a302d16" is_pre_election: true
I20260925 14:56:46.455768 21301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28eb0698b5c84c198cfa7abeb567d92b" candidate_uuid: "cf6bb71065f24ffc873651c0bd4d1870" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6bd2d980f49476cb844d04f3a302d16"
I20260925 14:56:46.455869 21301 raft_consensus.cc:3037] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:56:46.456341 21164 raft_consensus.cc:2454] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6bb71065f24ffc873651c0bd4d1870 in term 1.
I20260925 14:56:46.456501 21372 leader_election.cc:302] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aaaeafd77b84626a7564754bdec6e27, cf6bb71065f24ffc873651c0bd4d1870; no voters:
I20260925 14:56:46.456591 21509 raft_consensus.cc:2781] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:56:46.456733 21509 raft_consensus.cc:686] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 1 LEADER]: Becoming Leader. State: Replica: cf6bb71065f24ffc873651c0bd4d1870, State: Running, Role: LEADER
I20260925 14:56:46.456815 21301 raft_consensus.cc:2454] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6bb71065f24ffc873651c0bd4d1870 in term 1.
I20260925 14:56:46.456836 21509 consensus_queue.cc:237] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } }
I20260925 14:56:46.457590 20998 catalog_manager.cc:5495] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf6bb71065f24ffc873651c0bd4d1870 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "cf6bb71065f24ffc873651c0bd4d1870" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6bb71065f24ffc873651c0bd4d1870" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39255 } health_report { overall_health: HEALTHY } } }
I20260925 14:56:46.462915 21499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.460503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.9.141.67:39255)
0925 14:56:46.460550 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:56:46.460552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:56:46.460583 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:56:46.460655 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:56:46.460656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:56:46.460673 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:56:46.460755 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:56:46.460917 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.461371 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.462147 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:56:46.462348 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:56:46.462423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:56:46.462588 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:56:46.462688 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:56:46.462775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:56:46.462815 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 14:56:46.462832 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:56:46.464419 21301 raft_consensus.cc:1261] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 1 FOLLOWER]: Refusing update from remote peer cf6bb71065f24ffc873651c0bd4d1870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:46.464476 21164 raft_consensus.cc:1261] T 28eb0698b5c84c198cfa7abeb567d92b P 2aaaeafd77b84626a7564754bdec6e27 [term 1 FOLLOWER]: Refusing update from remote peer cf6bb71065f24ffc873651c0bd4d1870: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:56:46.464716 21509 consensus_queue.cc:1035] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aaaeafd77b84626a7564754bdec6e27" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 39333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:46.464841 21511 consensus_queue.cc:1035] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6bd2d980f49476cb844d04f3a302d16" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 45797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:56:46.473644 21514 mvcc.cc:205] Tried to move back new op lower bound from 7333266253675704320 to 7333266253647388672. Current Snapshot: MvccSnapshot[applied={T|T < 7333266253675704320}]
I20260925 14:56:46.474655 21515 mvcc.cc:205] Tried to move back new op lower bound from 7333266253675704320 to 7333266253647388672. Current Snapshot: MvccSnapshot[applied={T|T < 7333266253675704320}]
I20260925 14:56:46.475860 21516 mvcc.cc:205] Tried to move back new op lower bound from 7333266253675704320 to 7333266253647388672. Current Snapshot: MvccSnapshot[applied={T|T < 7333266253675704320}]
I20260925 14:56:46.477301 20997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:46.477427 20997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:56:46.478109 20997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 467583c4fb6040b0805fe249f2f90c41: Sending DeleteTablet for 3 replicas of tablet 28eb0698b5c84c198cfa7abeb567d92b
I20260925 14:56:46.478473 21416 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eb0698b5c84c198cfa7abeb567d92b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:46 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:53754
I20260925 14:56:46.478502 21282 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eb0698b5c84c198cfa7abeb567d92b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:46 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:39618
I20260925 14:56:46.478838 21144 tablet_service.cc:1514] Processing DeleteTablet for tablet 28eb0698b5c84c198cfa7abeb567d92b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:56:46 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:45552
I20260925 14:56:46.479439 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 21079
I20260925 14:56:46.482337 21524 tablet_replica.cc:330] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: stopping tablet replica
I20260925 14:56:46.482482 21524 raft_consensus.cc:2229] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:56:46.482632 21524 raft_consensus.cc:2258] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:46.483155 21524 ts_tablet_manager.cc:1905] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:46.484231 21525 tablet_replica.cc:330] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: stopping tablet replica
I20260925 14:56:46.484411 21525 raft_consensus.cc:2229] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:56:46.484573 21525 raft_consensus.cc:2258] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:56:46.485143 21525 ts_tablet_manager.cc:1905] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:56:46.485329 21524 ts_tablet_manager.cc:1918] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:46.485469 21524 log.cc:1196] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/28eb0698b5c84c198cfa7abeb567d92b
I20260925 14:56:46.485858 21524 ts_tablet_manager.cc:1939] T 28eb0698b5c84c198cfa7abeb567d92b P cf6bb71065f24ffc873651c0bd4d1870: Deleting consensus metadata
I20260925 14:56:46.487483 21525 ts_tablet_manager.cc:1918] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:56:46.488049 21525 log.cc:1196] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/28eb0698b5c84c198cfa7abeb567d92b
I20260925 14:56:46.488355 21525 ts_tablet_manager.cc:1939] T 28eb0698b5c84c198cfa7abeb567d92b P f6bd2d980f49476cb844d04f3a302d16: Deleting consensus metadata
I20260925 14:56:46.488394 20985 catalog_manager.cc:4841] TS cf6bb71065f24ffc873651c0bd4d1870 (127.9.141.67:39255): tablet 28eb0698b5c84c198cfa7abeb567d92b (table test-table [id=a0edd965856c41b5a5e517ea8362cc7d]) successfully deleted
W20260925 14:56:46.492365 20985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:39333 (error 108)
I20260925 14:56:46.492799 20983 catalog_manager.cc:4841] TS f6bd2d980f49476cb844d04f3a302d16 (127.9.141.66:45797): tablet 28eb0698b5c84c198cfa7abeb567d92b (table test-table [id=a0edd965856c41b5a5e517ea8362cc7d]) successfully deleted
I20260925 14:56:46.493556 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:56:46.493415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:39333 (local address 127.0.0.1:57868)
0925 14:56:46.493467 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:56:46.493522 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:39333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:56:46.493556 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 21218
W20260925 14:56:46.493721 20985 catalog_manager.cc:4584] TS 2aaaeafd77b84626a7564754bdec6e27 (127.9.141.65:39333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28eb0698b5c84c198cfa7abeb567d92b: Network error: Client connection negotiation failed: client connection to 127.9.141.65:39333: connect: Connection refused (error 111)
I20260925 14:56:46.500653 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 21353
I20260925 14:56:46.506349 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 20966
I20260925 14:56:46.511703 9781 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:56:46.856015 9781 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:56:46.856119 9781 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:56:47.200544 9781 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:56:47.200 UTC [20552] LOG: received smart shutdown request
2026-09-25 14:56:47.204 UTC [20552] LOG: background worker "logical replication launcher" (PID 20561) exited with exit code 1
2026-09-25 14:56:47.204 UTC [20556] LOG: shutting down
2026-09-25 14:56:47.384 UTC [20552] LOG: database system is shut down
2026-09-25T14:56:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46466 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:56:47 dist-test-slave-1sv7 krb5kdc[21536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:56:47 dist-test-slave-1sv7 krb5kdc[21536](info): set up 2 sockets
Sep 25 14:56:47 dist-test-slave-1sv7 krb5kdc[21536](info): commencing operation
krb5kdc: starting...
W20260925 14:56:48.544451 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:56:48 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:56:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/rangeradmin_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/rangeradmin_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/rangerlookup_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/rangerlookup_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
I20260925 14:56:48.712460 9781 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task5afOW1/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/postgres -l logfile start
2026-09-25 14:56:51.362 UTC [21566] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:56:51.362 UTC [21566] LOG: listening on IPv4 address "127.9.141.84", port 56389
2026-09-25 14:56:51.366 UTC [21566] LOG: listening on Unix socket "/tmp/.s.PGSQL.56389"
2026-09-25 14:56:51.401 UTC [21569] LOG: database system was shut down at 2026-09-25 14:56:49 UTC
2026-09-25 14:56:51.405 UTC [21566] LOG: database system is ready to accept connections
I20260925 14:56:52.380209 9781 mini_postgres.cc:96] Postgres bound to 56389
2026-09-25 14:56:52.386 UTC [21577] FATAL: database "slave" does not exist
127.9.141.84:56389 - accepting connections
I20260925 14:56:52.386756 9781 mini_ranger.cc:161] Starting Ranger...
I20260925 14:56:52.401017 9781 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:56:52.913666 9781 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:56:52.913748 9781 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-admin
2026-09-25 14:56:53,199 [I] DB FLAVOR :POSTGRES
2026-09-25 14:56:53,199 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:56:53,199 [I] Checking connection..
2026-09-25 14:56:53,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:56:53,455 [I] Checking connection passed.
2026-09-25 14:56:53,455 [I] --------- Verifying version history table ---------
2026-09-25 14:56:53,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:56:53,714 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:56:53,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:56:53,963 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:56:53,963 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:56:53,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:56:54.205 UTC [21641] WARNING: there is no transaction in progress
2026-09-25 14:56:54,211 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:56:54,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:56:54,476 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:56:54,477 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:56:54,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:56:54,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:56:55,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1sv7.c.gcp-upstream.internal','N') ;"
2026-09-25 14:56:55,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:56:55,522 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:56:55,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:56:55,773 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:56:55,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:56:56,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:56:56,248 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:56:56,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:56:57.256 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.275 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.292 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.309 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.325 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.343 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.400 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.406 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.420 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.431 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.441 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.456 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.466 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.473 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.480 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:57.487 UTC [21749] WARNING: there is no transaction in progress
2026-09-25 14:56:58,197 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:56:58,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:56:58,418 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:56:58,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:56:58,689 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:56:58,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:56:58,951 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:56:58,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:56:59,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:56:59,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1sv7.c.gcp-upstream.internal';"
2026-09-25 14:56:59,692 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:56:59,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:56:59,956 [I] DB_PATCHES have already been applied
I20260925 14:56:59.961840 9781 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-admin:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task5afOW1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:56:59.961901 9781 mini_ranger.cc:193] Using host: 127.9.141.84
I20260925 14:56:59.964993 9781 mini_ranger.cc:227] Ranger admin URL: http://127.9.141.84:56297
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:57:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:57:01.079919 9781 mini_ranger.cc:160] Time spent starting Ranger: real 8.693s user 0.000s sys 0.009s
Sep 25, 2026 2:57:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:57:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:57:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:57:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:57:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:57:23.161033 9781 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.141.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.141.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/rangerkms_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.141.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/rangerkms_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.141.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.141.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.141.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/HTTP_127.9.141.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:57:23 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348243, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:57:23.248835 9781 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:57:23.258195 9781 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:57:24.268205 9781 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:57:24.268276 9781 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms
I20260925 14:57:24.268289 9781 mini_ranger_kms.cc:227] Using postgres at 127.9.141.84:56389
2026-09-25 14:57:24,433 [I] DB FLAVOR :POSTGRES
2026-09-25 14:57:24,433 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:57:24,433 [I] Checking connection
2026-09-25 14:57:24,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:57:24,692 [I] connection success
2026-09-25 14:57:24,693 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:57:24,693 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:57:24,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:57:24,950 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:57:24,950 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:57:24,950 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:57:24,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.141.84:56389/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:57:25,199 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:57:25.699528 9781 mini_ranger_kms.cc:310] Created kms service
I20260925 14:57:25.916968 9781 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:57:25.961723 9781 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:57:25 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348243, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:26.366186 9781 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:57:26.366446 9781 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms:/tmp/dist-test-task5afOW1/build/release/bin/postgresql.jar:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task5afOW1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:57:26.366474 9781 mini_ranger_kms.cc:240] Using host: 127.9.141.84
I20260925 14:57:26.370332 9781 mini_ranger_kms.cc:276] Ranger KMS PID: 21929
I20260925 14:57:26.370414 9781 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.141.84:59421
Sep 25, 2026 2:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:57:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:57:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:57:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 14:57:28.691992 9781 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.443s user 0.020s sys 0.128s
I20260925 14:57:28.692106 9781 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:57:28.692154 9781 mini_ranger_kms.cc:401] 127.9.141.84:59421/kms/v1/keys
Sep 25 14:57:28 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348248, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.141.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 14:57:29 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348248, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.141.84@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
Sep 25 14:57:29 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348248, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.141.84@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
Sep 25 14:57:31 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:57:31.823319 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41499
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:41499
--ranger_config_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:31.907663 21980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:31.907917 21980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:31.907951 21980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:31.909126 21980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:31.909191 21980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:31.909216 21980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:31.909240 21980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:31.909282 21980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:31.910841 21980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.141.84:59421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:41499
--ranger_config_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:31.911115 21980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:31.911334 21980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:31.916324 21986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.916424 21980 server_base.cc:1034] running on GCE node
W20260925 14:57:31.916333 21985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:31.916330 21988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:31.916795 21980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:31.917092 21980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:31.918247 21980 hybrid_clock.cc:648] HybridClock initialized: now 1790348251918247 us; error 49 us; skew 500 ppm
Sep 25 14:57:31 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348251, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:31.927000 21980 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:31.927531 21980 webserver.cc:460] Webserver started at http://127.9.141.126:34421/ using document root <none> and password file <none>
I20260925 14:57:31.927774 21980 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:31.927838 21980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:31.928000 21980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:57:31 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348251, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:31.984078 21980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "444d771a1ae24584ae8cf91e8990fd01"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-1sv7"
server_key: "40e217f7321c868513a387304868e7bd"
server_key_iv: "6744640a0abb6a3b10b8ba150603b4fc"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:31.984545 21980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "444d771a1ae24584ae8cf91e8990fd01"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-1sv7"
server_key: "40e217f7321c868513a387304868e7bd"
server_key_iv: "6744640a0abb6a3b10b8ba150603b4fc"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:31.986006 21980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:32.002111 21995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:32.002317 21980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:32.002403 21980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "444d771a1ae24584ae8cf91e8990fd01"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-1sv7"
server_key: "40e217f7321c868513a387304868e7bd"
server_key_iv: "6744640a0abb6a3b10b8ba150603b4fc"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:32.002507 21980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:32.016709 21980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:32.018936 21980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:32.019089 21980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:32.023985 21980 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:41499
I20260925 14:57:32.024014 22047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:41499 every 8 connection(s)
I20260925 14:57:32.024415 21980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:32.025218 22048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:32.028342 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Bootstrap starting.
I20260925 14:57:32.029093 22048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:32.029446 22048 log.cc:824] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:32.029630 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 21980
I20260925 14:57:32.029811 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:57:32.030026 9781 external_mini_cluster.cc:1428] Setting key 6ac83ddd1836acaf3989ad1a6242cd97
I20260925 14:57:32.030287 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: No bootstrap required, opened a new log
I20260925 14:57:32.033406 22048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:32.033592 22048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:32.033623 22048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444d771a1ae24584ae8cf91e8990fd01, State: Initialized, Role: FOLLOWER
I20260925 14:57:32.033771 22048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:32.033862 22048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:32.033910 22048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:32.033964 22048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:32.034688 22048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:32.034802 22048 leader_election.cc:302] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 444d771a1ae24584ae8cf91e8990fd01; no voters:
I20260925 14:57:32.035005 22048 leader_election.cc:288] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:32.035072 22053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:32.035295 22053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 LEADER]: Becoming Leader. State: Replica: 444d771a1ae24584ae8cf91e8990fd01, State: Running, Role: LEADER
I20260925 14:57:32.035333 22048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:32.035461 22053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:32.035784 22053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 444d771a1ae24584ae8cf91e8990fd01. Latest consensus state: current_term: 1 leader_uuid: "444d771a1ae24584ae8cf91e8990fd01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } } }
I20260925 14:57:32.035841 22053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:32.036067 22060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:32.036886 22060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:32.037257 22048 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:57:32.037678 22054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "444d771a1ae24584ae8cf91e8990fd01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } } }
I20260925 14:57:32.037838 22054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:32.038643 22070 mvcc.cc:205] Tried to move back new op lower bound from 7333266440347127808 to 7333266440337530880. Current Snapshot: MvccSnapshot[applied={T|T < 7333266440347127808}]
Sep 25 14:57:32 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:32.044714 22060 catalog_manager.cc:1347] Generated new cluster ID: 2bbf3e8abd0c4667937e625d9bc9e865
I20260925 14:57:32.044893 22060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:32.048468 22052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.030592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.9.141.126:41499)
0925 14:57:32.030789 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:32.030799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:32.030832 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:57:32.033402 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:32.033411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:32.033476 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:32.033700 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:32.034624 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.036270 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.037514 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.038242 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.038358 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.042446 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:32.042504 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:32.042504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:32.042639 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:32.046095 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.046510 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.046513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.046514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.046650 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.046853 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.046854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.046854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.047237 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:32.047377 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:32.047887 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 14:57:32.048150 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260925 14:57:32.055536 22060 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:32.056449 22060 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:32.065586 22060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Generated new TSK 0
I20260925 14:57:32.065845 22060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:57:32.780139 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41499
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:32.878335 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:32.878576 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:32.878618 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:32.879813 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:32.879891 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:32.879937 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:57:32.881525 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.141.84:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:32.881785 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:32.882030 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:32.887434 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:32.887471 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:32.887471 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:32.887923 22093 server_base.cc:1034] running on GCE node
I20260925 14:57:32.888137 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:32.888451 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:32.889642 22093 hybrid_clock.cc:648] HybridClock initialized: now 1790348252889624 us; error 36 us; skew 500 ppm
Sep 25 14:57:32 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:32.892287 22093 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:57:32.892813 22093 webserver.cc:460] Webserver started at http://127.9.141.65:36939/ using document root <none> and password file <none>
I20260925 14:57:32.893126 22093 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:32.893198 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:32.893337 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:57:32 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:32.905061 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "8b90284652c24819843580bab1b2e65a"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-1sv7"
server_key: "edb5eec567e73f4fed48fc24e18f7095"
server_key_iv: "ad7ff091d70affa394fbaa47e4013188"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:32.905542 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "8b90284652c24819843580bab1b2e65a"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-1sv7"
server_key: "edb5eec567e73f4fed48fc24e18f7095"
server_key_iv: "ad7ff091d70affa394fbaa47e4013188"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:32.912801 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260925 14:57:32.927175 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:32.927486 22093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:32.927584 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "8b90284652c24819843580bab1b2e65a"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-1sv7"
server_key: "edb5eec567e73f4fed48fc24e18f7095"
server_key_iv: "ad7ff091d70affa394fbaa47e4013188"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:32.927673 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:32.941172 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:32.943625 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:32.943849 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:32.944154 22093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:32.944574 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:32.944664 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:32.944756 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:32.944826 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:32.951181 22093 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:45373
I20260925 14:57:32.951678 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:32.952395 22223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:45373 every 8 connection(s)
I20260925 14:57:32.959352 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22093
I20260925 14:57:32.959489 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:32.959685 9781 external_mini_cluster.cc:1428] Setting key c79fc4ef4dcd1565c762d60ecba55abf
Sep 25 14:57:32 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348252, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:32.964504 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.953302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41499 (local address 127.9.141.65:42203)
0925 14:57:32.953736 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:32.953785 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:32.954424 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:32.954877 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:32.954882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:32.955149 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:32.955429 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:32.955436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.956676 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.956677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:32.957965 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:32.957969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.958391 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.958392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:32.958540 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.959134 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:32.959145 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:32.960771 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:32.962716 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:32.962723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:32.962726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:32.963060 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:32.963272 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:32.963273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:32.963273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:32.963417 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:32.963745 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:32.963751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:32.963918 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 14:57:32.964025 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:57:32.964193 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":76,"threads_started":1}
I20260925 14:57:32.964861 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:32.953589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:42203 (local address 127.9.141.126:41499)
0925 14:57:32.954617 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:32.954619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:32.954627 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:32.954654 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:32.954655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:32.954666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:32.954733 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:32.955545 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.956574 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.958088 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:32.958284 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:32.958349 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:32.960879 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:32.960880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:32.960881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:32.960889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:32.962612 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.963145 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.963146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.963146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.963209 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:32.963485 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:32.963486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:32.963487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:32.963632 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:32.964627 (+ 995us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:32.964677 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:57:32.964769 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":74,"threads_started":1}
I20260925 14:57:32.965368 22224 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41499
I20260925 14:57:32.965500 22224 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:32.965727 22224 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:32.966377 22012 ts_manager.cc:194] Registered new tserver with Master: 8b90284652c24819843580bab1b2e65a (127.9.141.65:45373)
I20260925 14:57:32.967332 22012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:42203
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:57:32.991660 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41499
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:33.100056 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:33.100339 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:33.100414 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:33.102177 22230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:33.102254 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:33.102298 22230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:57:33.104542 22230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.141.84:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:33.104784 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:33.105103 22230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:33.111455 22238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.111477 22240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.111812 22237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:33.112134 22230 server_base.cc:1034] running on GCE node
I20260925 14:57:33.112337 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:33.112617 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:33.113801 22230 hybrid_clock.cc:648] HybridClock initialized: now 1790348253113767 us; error 34 us; skew 500 ppm
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:33.117074 22230 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:57:33.117573 22230 webserver.cc:460] Webserver started at http://127.9.141.66:32995/ using document root <none> and password file <none>
I20260925 14:57:33.117877 22230 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:33.117969 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:33.118125 22230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:33.132366 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "3b30788779fa4f7098359a179eea60d3"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "a1e6402a8865f0be65a6f480ad07a6ac"
server_key_iv: "6a0296723b9f71ce6c14bc8ca205e848"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.132927 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "3b30788779fa4f7098359a179eea60d3"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "a1e6402a8865f0be65a6f480ad07a6ac"
server_key_iv: "6a0296723b9f71ce6c14bc8ca205e848"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.135895 22230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:33.155720 22247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.155961 22230 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260925 14:57:33.156059 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "3b30788779fa4f7098359a179eea60d3"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "a1e6402a8865f0be65a6f480ad07a6ac"
server_key_iv: "6a0296723b9f71ce6c14bc8ca205e848"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.156154 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:33.192202 22230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:33.194818 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:33.195060 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:33.195298 22230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:33.195668 22230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:33.195719 22230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.195749 22230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:33.195817 22230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.202273 22230 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:39589
I20260925 14:57:33.202683 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:33.208009 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22230
I20260925 14:57:33.208117 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:33.208316 9781 external_mini_cluster.cc:1428] Setting key 8bcc6a00a24fda944f8cdeaa872d8c86
I20260925 14:57:33.213529 22360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:39589 every 8 connection(s)
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:33.245462 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:33.221297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:54883 (local address 127.9.141.126:41499)
0925 14:57:33.221364 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:33.221367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:33.224701 (+ 3334us) server_negotiation.cc:407] Connection header received
0925 14:57:33.226857 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:33.226859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:33.226873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:33.226903 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:33.228728 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.229518 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.232585 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.232918 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.232982 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:33.237291 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:33.237292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:33.237293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:33.237300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:33.239071 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:33.239557 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:33.239558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:33.239558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:33.239616 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:33.244534 (+ 4918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:33.244536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:33.244536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:33.244697 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:33.244755 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:33.245237 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 14:57:33.245333 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:57:33.246377 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:33.221080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41499 (local address 127.9.141.66:54883)
0925 14:57:33.224524 (+ 3444us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:33.224583 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:33.226711 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:33.227118 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:33.227122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:33.227398 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:33.227719 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:33.227726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.229641 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.229642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:33.230980 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:33.230983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.233096 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.233097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:33.233245 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:33.235381 (+ 2136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:33.235393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:33.237171 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:33.239168 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:33.239175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:33.239177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:33.239465 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:33.239682 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:33.239683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:33.239683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:33.239805 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:33.244854 (+ 5049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:33.244863 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:33.245062 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 14:57:33.245904 (+ 842us) client_negotiation.cc:240] Negotiation successful
0925 14:57:33.246091 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3385,"thread_start_us":252,"threads_started":1}
I20260925 14:57:33.247116 22361 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41499
I20260925 14:57:33.247215 22361 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:33.247531 22361 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:33.247953 22012 ts_manager.cc:194] Registered new tserver with Master: 3b30788779fa4f7098359a179eea60d3 (127.9.141.66:39589)
I20260925 14:57:33.248570 22012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:54883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:57:33.253692 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41499
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:33.386385 22367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:33.386760 22367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:33.386850 22367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:33.388681 22367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:33.388803 22367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:33.388890 22367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:57:33.391201 22367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.141.84:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:33.391703 22367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:33.392033 22367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:33.398954 22375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.399195 22372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:33.400139 22367 server_base.cc:1034] running on GCE node
W20260925 14:57:33.400466 22373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:33.400779 22367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:33.401062 22367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:33.408551 22367 hybrid_clock.cc:648] HybridClock initialized: now 1790348253408524 us; error 34 us; skew 500 ppm
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:33.411947 22367 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:57:33.412568 22367 webserver.cc:460] Webserver started at http://127.9.141.67:38889/ using document root <none> and password file <none>
I20260925 14:57:33.412957 22367 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:33.413086 22367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:33.413278 22367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:33.427546 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "7d00bb91928642aba55224edc9f205e1"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "c0773b3a203577013bc199ff97bfd08a"
server_key_iv: "f8ed0a7a0dff2fdb360efd440ff3c170"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.428015 22367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "7d00bb91928642aba55224edc9f205e1"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "c0773b3a203577013bc199ff97bfd08a"
server_key_iv: "f8ed0a7a0dff2fdb360efd440ff3c170"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.429952 22367 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:57:33.444362 22382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.444586 22367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:33.444689 22367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "7d00bb91928642aba55224edc9f205e1"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "c0773b3a203577013bc199ff97bfd08a"
server_key_iv: "f8ed0a7a0dff2fdb360efd440ff3c170"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.444792 22367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:33.476058 22367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:33.478818 22367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:33.478983 22367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:33.479281 22367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:33.479650 22367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:33.479698 22367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.479729 22367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:33.479753 22367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.486996 22367 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39655
I20260925 14:57:33.487437 22367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:33.494954 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22367
I20260925 14:57:33.495066 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:33.495249 9781 external_mini_cluster.cc:1428] Setting key ea5d11100a1f5d2b11ebb3d5bd95faa0
I20260925 14:57:33.496999 22495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39655 every 8 connection(s)
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:33.515240 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:33.497996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:53497 (local address 127.9.141.126:41499)
0925 14:57:33.498044 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:33.498046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:33.500714 (+ 2668us) server_negotiation.cc:407] Connection header received
0925 14:57:33.501425 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:33.501426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:33.501438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:33.501470 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:33.502378 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.503149 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.504716 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.504995 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.505044 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:33.509758 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:33.509759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:33.509759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:33.509768 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:33.511580 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:33.514036 (+ 2456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:33.514037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:33.514037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:33.514106 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:33.514421 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:33.514421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:33.514422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:33.514569 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:33.514628 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:33.515038 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 14:57:33.515140 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:57:33.515810 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:33.497691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41499 (local address 127.9.141.67:53497)
0925 14:57:33.500560 (+ 2869us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:33.500621 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:33.501299 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:33.501688 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:33.501693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:33.501965 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:33.502261 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:33.502267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.503274 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.503275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:33.504611 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:33.504614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.506562 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.506563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:33.506802 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:33.507778 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:33.507789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:33.509640 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:33.513564 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:33.513573 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:33.513576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:33.513911 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:33.514183 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:33.514183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:33.514184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:33.514336 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:33.514665 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:33.514671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:33.514853 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 14:57:33.515388 (+ 535us) client_negotiation.cc:240] Negotiation successful
0925 14:57:33.515566 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2792,"thread_start_us":98,"threads_started":1}
I20260925 14:57:33.516345 22496 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41499
I20260925 14:57:33.516523 22496 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:33.516743 22496 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:33.517133 22012 ts_manager.cc:194] Registered new tserver with Master: 7d00bb91928642aba55224edc9f205e1 (127.9.141.67:39655)
I20260925 14:57:33.517647 22012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:53497
I20260925 14:57:33.519095 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:33.519675 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22093
I20260925 14:57:33.525990 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22230
I20260925 14:57:33.530519 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22367
I20260925 14:57:33.536331 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 21980
I20260925 14:57:33.553171 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41499
--webserver_interface=127.9.141.126
--webserver_port=34421
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:41499
--ranger_config_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:33.635059 22499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:33.635279 22499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:33.635303 22499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:33.636500 22499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:33.636540 22499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:33.636554 22499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:33.636566 22499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:33.636579 22499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:33.638232 22499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:41499
--ranger_config_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=34421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:33.638468 22499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:33.638698 22499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:33.643600 22505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.643745 22504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.643610 22507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:33.643999 22499 server_base.cc:1034] running on GCE node
I20260925 14:57:33.644282 22499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:33.644609 22499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:33.645767 22499 hybrid_clock.cc:648] HybridClock initialized: now 1790348253645747 us; error 37 us; skew 500 ppm
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:33.648245 22499 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:33.648748 22499 webserver.cc:460] Webserver started at http://127.9.141.126:34421/ using document root <none> and password file <none>
I20260925 14:57:33.648981 22499 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:33.649034 22499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:33.711056 22499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:33.711864 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.712050 22499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:33.712160 22499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "444d771a1ae24584ae8cf91e8990fd01"
format_stamp: "Formatted at 2026-09-25 14:57:31 on dist-test-slave-1sv7"
server_key: "40e217f7321c868513a387304868e7bd"
server_key_iv: "6744640a0abb6a3b10b8ba150603b4fc"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.712635 22499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:33.738545 22499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:33.741065 22499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:33.741235 22499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:33.746129 22499 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:41499
I20260925 14:57:33.746197 22567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:41499 every 8 connection(s)
I20260925 14:57:33.746592 22499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:33.747862 22568 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:57:33.748581 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Bootstrap starting.
I20260925 14:57:33.749131 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22499
I20260925 14:57:33.749449 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:45373
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:41499
--webserver_port=36939
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:33.759430 22568 log.cc:824] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:33.762034 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:33.762352 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Bootstrap complete.
I20260925 14:57:33.765748 22568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:33.766077 22568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444d771a1ae24584ae8cf91e8990fd01, State: Initialized, Role: FOLLOWER
I20260925 14:57:33.766239 22568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:33.766306 22568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:33.766343 22568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:33.766388 22568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:57:33.767761 22568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:33.767870 22568 leader_election.cc:302] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 444d771a1ae24584ae8cf91e8990fd01; no voters:
I20260925 14:57:33.768033 22568 leader_election.cc:288] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:57:33.768141 22572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:57:33.768276 22568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:33.768297 22572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [term 2 LEADER]: Becoming Leader. State: Replica: 444d771a1ae24584ae8cf91e8990fd01, State: Running, Role: LEADER
I20260925 14:57:33.768419 22572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } }
I20260925 14:57:33.768651 22572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 444d771a1ae24584ae8cf91e8990fd01. Latest consensus state: current_term: 2 leader_uuid: "444d771a1ae24584ae8cf91e8990fd01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } } }
I20260925 14:57:33.768751 22572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:33.768651 22573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "444d771a1ae24584ae8cf91e8990fd01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444d771a1ae24584ae8cf91e8990fd01" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41499 } } }
I20260925 14:57:33.768810 22573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:33.769735 22577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:33.770277 22577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:33.770411 22577 catalog_manager.cc:1259] Loaded cluster ID: 2bbf3e8abd0c4667937e625d9bc9e865
I20260925 14:57:33.770447 22577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:33.770656 22568 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:57:33.774362 22577 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:33.774914 22577 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Loaded TSK: 0
I20260925 14:57:33.775218 22577 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:57:33.847105 22570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:33.847433 22570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:33.847481 22570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:33.849521 22570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:33.849598 22570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:33.849649 22570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:57:33.852273 22570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:45373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=36939
--tserver_master_addrs=127.9.141.126:41499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:33.852594 22570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:33.852906 22570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:33.859915 22608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.859925 22606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:33.860172 22605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:33.860697 22570 server_base.cc:1034] running on GCE node
I20260925 14:57:33.860939 22570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:33.861279 22570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:33.862452 22570 hybrid_clock.cc:648] HybridClock initialized: now 1790348253862441 us; error 31 us; skew 500 ppm
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:33.865293 22570 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:57:33.865859 22570 webserver.cc:460] Webserver started at http://127.9.141.65:36939/ using document root <none> and password file <none>
I20260925 14:57:33.866097 22570 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:33.866194 22570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:33.909242 22570 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.000s sys 0.001s
I20260925 14:57:33.910286 22616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.910447 22570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:33.910538 22570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "8b90284652c24819843580bab1b2e65a"
format_stamp: "Formatted at 2026-09-25 14:57:32 on dist-test-slave-1sv7"
server_key: "edb5eec567e73f4fed48fc24e18f7095"
server_key_iv: "ad7ff091d70affa394fbaa47e4013188"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:33.911028 22570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:33.926787 22570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:33.929170 22570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:33.929327 22570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:33.929590 22570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:33.929943 22570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:33.930006 22570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.930038 22570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:33.930055 22570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:33.937143 22570 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:45373
I20260925 14:57:33.937628 22570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:33.939332 22729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:45373 every 8 connection(s)
I20260925 14:57:33.946806 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22570
I20260925 14:57:33.947126 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:39589
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:41499
--webserver_port=32995
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:57:33 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:33.957791 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:33.939722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:33115 (local address 127.9.141.126:41499)
0925 14:57:33.939932 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:33.939941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:33.945316 (+ 5375us) server_negotiation.cc:407] Connection header received
0925 14:57:33.945490 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:33.945497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:33.945554 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:33.945702 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:33.946445 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.947911 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.949439 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.950083 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.950199 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:33.953463 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:33.953468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:33.953469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:33.953481 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:33.955549 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:33.956271 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:33.956274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:33.956275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:33.956404 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:33.956691 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:33.956692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:33.956692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:33.956988 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:33.957056 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:33.957413 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 14:57:33.957537 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260925 14:57:33.958261 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:33.939377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41499 (local address 127.9.141.65:33115)
0925 14:57:33.944520 (+ 5143us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:33.944588 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:33.945274 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:33.945777 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:33.945781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:33.946030 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:33.946304 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:33.946310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.948042 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.948044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:33.949339 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:33.949343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:33.950209 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:33.950211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:33.950347 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:33.951178 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:33.951188 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:33.953082 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:33.955872 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:33.955879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:33.955882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:33.956172 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:33.956476 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:33.956477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:33.956477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:33.956617 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:33.957094 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:33.957100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:33.957241 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:57:33.957898 (+ 657us) client_negotiation.cc:240] Negotiation successful
0925 14:57:33.958056 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5089,"thread_start_us":81,"threads_started":1}
I20260925 14:57:33.959033 22730 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41499
I20260925 14:57:33.959134 22730 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:33.959350 22730 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
W20260925 14:57:33.960208 22730 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:41499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:57:33.960282 22730 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:33.960346 22730 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:33.960676 22730 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:33.960753 22730 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
W20260925 14:57:33.960995 22730 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:57:34.070519 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:34.070802 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:34.070838 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:34.072572 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:34.072629 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:34.072669 22734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:57:34.074769 22734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:39589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=32995
--tserver_master_addrs=127.9.141.126:41499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:34.075053 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:34.075330 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:34.081585 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.081681 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.081605 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:34.082399 22734 server_base.cc:1034] running on GCE node
I20260925 14:57:34.082603 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:34.082856 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:34.090312 22734 hybrid_clock.cc:648] HybridClock initialized: now 1790348254090218 us; error 114 us; skew 500 ppm
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:34.093616 22734 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:57:34.094156 22734 webserver.cc:460] Webserver started at http://127.9.141.66:32995/ using document root <none> and password file <none>
I20260925 14:57:34.094399 22734 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:34.094458 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:34.110356 22734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:34.111290 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.111456 22734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:34.111547 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "3b30788779fa4f7098359a179eea60d3"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "a1e6402a8865f0be65a6f480ad07a6ac"
server_key_iv: "6a0296723b9f71ce6c14bc8ca205e848"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:34.111935 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:34.129235 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:34.131634 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:34.131796 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:34.132066 22734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:34.132454 22734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:34.132503 22734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.132539 22734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:34.132566 22734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.143289 22734 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:39589
I20260925 14:57:34.143775 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:34.144764 22863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:39589 every 8 connection(s)
I20260925 14:57:34.145641 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22734
I20260925 14:57:34.145951 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.141.84:59421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:39655
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:41499
--webserver_port=38889
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:34.158092 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.145413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:41123 (local address 127.9.141.126:41499)
0925 14:57:34.145494 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.145497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.148731 (+ 3234us) server_negotiation.cc:407] Connection header received
0925 14:57:34.149411 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.149413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.149431 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.149511 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.150309 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.150888 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.152013 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.152220 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.152293 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.154908 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.154909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.154910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:34.154918 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.156619 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:34.157095 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:34.157095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:34.157096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:34.157146 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:34.157405 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:34.157406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:34.157407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:34.157524 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.157597 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.157927 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.158025 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:57:34.158596 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.145420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41499 (local address 127.9.141.66:41123)
0925 14:57:34.148523 (+ 3103us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:34.148578 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:34.149266 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:34.149623 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:34.149628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:34.149865 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:34.150195 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:34.150201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.150993 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.150995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:34.151914 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:34.151917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.152337 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.152339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:34.152515 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.153088 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:34.153097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:34.154793 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:34.156742 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:34.156748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:34.156751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:34.156974 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:34.157216 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:34.157217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:34.157218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:34.157334 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:34.157599 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:34.157606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:34.157748 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 14:57:34.158261 (+ 513us) client_negotiation.cc:240] Negotiation successful
0925 14:57:34.158418 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3052,"thread_start_us":80,"threads_started":1}
I20260925 14:57:34.159402 22864 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41499
I20260925 14:57:34.159523 22864 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.159718 22864 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
W20260925 14:57:34.160354 22864 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:41499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:57:34.160471 22864 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.160544 22864 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:34.160841 22864 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.160914 22864 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
W20260925 14:57:34.161463 22864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:57:34.269326 22867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:34.269608 22867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:34.269647 22867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:34.271369 22867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:34.271432 22867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:34.271473 22867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:57:34.273694 22867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.141.84:59421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:39655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=38889
--tserver_master_addrs=127.9.141.126:41499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:34.273948 22867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:34.274230 22867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:34.280776 22875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.280795 22873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:34.280864 22872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:34.282670 22867 server_base.cc:1034] running on GCE node
I20260925 14:57:34.282886 22867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:34.283155 22867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:34.284343 22867 hybrid_clock.cc:648] HybridClock initialized: now 1790348254284294 us; error 33 us; skew 500 ppm
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:34.286870 22867 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:57:34.287421 22867 webserver.cc:460] Webserver started at http://127.9.141.67:38889/ using document root <none> and password file <none>
I20260925 14:57:34.287680 22867 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:34.287763 22867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:34.304822 22867 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 14:57:34.309728 22884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.309939 22867 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260925 14:57:34.310021 22867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "7d00bb91928642aba55224edc9f205e1"
format_stamp: "Formatted at 2026-09-25 14:57:33 on dist-test-slave-1sv7"
server_key: "c0773b3a203577013bc199ff97bfd08a"
server_key_iv: "f8ed0a7a0dff2fdb360efd440ff3c170"
server_key_version: "kuduclusterkey@0"
I20260925 14:57:34.310528 22867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:34.344270 22867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:34.346410 22867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:34.346550 22867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:34.346777 22867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:34.347098 22867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:34.347138 22867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.347167 22867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:34.347184 22867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:34.353926 22867 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39655
I20260925 14:57:34.354151 22997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39655 every 8 connection(s)
I20260925 14:57:34.354380 22867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:34.354777 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 22867
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:34.371567 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.355288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.9.141.126:41499)
0925 14:57:34.356293 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.356295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.356302 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:34.356319 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.356319 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.356329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.356412 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.357221 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.357915 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.359036 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.359360 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.359422 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.361033 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.361034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.361035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:34.361042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.368861 (+ 7819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:34.370476 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:34.370477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:34.370478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:34.370536 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:34.370928 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:34.370928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:34.370929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:34.371057 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.371248 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.371285 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.371376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":54,"threads_started":1}
I20260925 14:57:34.371938 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.355222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:50749 (local address 127.9.141.126:41499)
0925 14:57:34.355272 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:34.355274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:34.355520 (+ 246us) server_negotiation.cc:407] Connection header received
0925 14:57:34.356194 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:34.356195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:34.356207 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:34.356274 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:34.357067 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.357867 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.359319 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.359560 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.359599 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.368424 (+ 8825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:34.368426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:34.368427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:34.368434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:34.370183 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:34.370768 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:34.370769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:34.370770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:34.370824 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:34.371216 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:34.371217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:34.371217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:34.371334 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:34.371376 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:34.371764 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 14:57:34.371855 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:34.372571 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:34.355224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41499 (local address 127.9.141.67:50749)
0925 14:57:34.355393 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:34.355445 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:34.356085 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:34.356428 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:34.356433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:34.356685 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:34.356959 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:34.356964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.358341 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.358342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:34.359224 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:34.359227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:34.359714 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:34.359717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:34.359853 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:34.360299 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:34.360309 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:34.368249 (+ 7940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:34.370273 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:34.370279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:34.370282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:34.370692 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:34.370910 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:34.370911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:34.370911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:34.371066 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:34.371425 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:34.371431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:34.371603 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 14:57:34.372116 (+ 513us) client_negotiation.cc:240] Negotiation successful
0925 14:57:34.372301 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260925 14:57:34.373034 22998 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41499
I20260925 14:57:34.373116 22998 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.373262 22998 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
W20260925 14:57:34.373999 22998 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:41499 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:57:34.374078 22998 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.374145 22998 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:34.374491 22998 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.374569 22998 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
W20260925 14:57:34.374895 22998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:57:34 dist-test-slave-1sv7 krb5kdc[21536](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:34.961171 22730 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:34.961326 22730 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:34.961699 22532 ts_manager.cc:194] Registered new tserver with Master: 8b90284652c24819843580bab1b2e65a (127.9.141.65:45373)
I20260925 14:57:34.962453 22532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:33115
I20260925 14:57:35.161645 22864 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:35.161823 22864 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:35.162093 22532 ts_manager.cc:194] Registered new tserver with Master: 3b30788779fa4f7098359a179eea60d3 (127.9.141.66:39589)
I20260925 14:57:35.162546 22532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:41123
Sep 25 14:57:35 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
I20260925 14:57:35.375072 22998 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:35.375222 22998 heartbeater.cc:511] Master 127.9.141.126:41499 requested a full tablet report, sending...
I20260925 14:57:35.375527 22532 ts_manager.cc:194] Registered new tserver with Master: 7d00bb91928642aba55224edc9f205e1 (127.9.141.67:39655)
I20260925 14:57:35.376039 22532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:50749
I20260925 14:57:35.385740 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:35.393142 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.386711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.9.141.126:41499)
0925 14:57:35.386966 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.386967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.386973 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:57:35.386991 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.386992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.387001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.387080 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:35.387619 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.388127 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.388913 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.389121 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.389213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.390453 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:35.390454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:35.390454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:35.390461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:35.391940 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:35.392302 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:35.392304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:35.392304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:35.392364 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:35.392627 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:35.392628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:35.392628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:35.392735 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:35.392844 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.392940 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.393020 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":57,"threads_started":1}
I20260925 14:57:35.395027 22532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:35.397217 22532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:35.403880 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.399908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39655 (local address 127.0.0.1:33686)
0925 14:57:35.400595 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:35.400637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:35.400730 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:35.401019 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:35.401021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:35.401023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:35.401070 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.401071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.401908 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.401909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.402882 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.402884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.403584 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.403586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.403661 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.403673 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:57:35.403753 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:57:35.403805 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":80,"threads_started":1}
I20260925 14:57:35.403882 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.400162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.9.141.67:39655)
0925 14:57:35.400305 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.400310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.400693 (+ 383us) server_negotiation.cc:407] Connection header received
0925 14:57:35.400852 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.400854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.400869 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.400925 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:35.401146 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.401821 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.402974 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.403482 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.403580 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.403640 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.403764 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.403806 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260925 14:57:35.404935 22914 tablet_service.cc:1467] Processing CreateTablet for tablet fbb19222a1714f62906c3ca15879287a (DEFAULT_TABLE table=test-table [id=8d4a5877e6b74c9cbf3d548d36553346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:35.405311 22914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb19222a1714f62906c3ca15879287a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:35.408123 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.400420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:39589 (local address 127.0.0.1:58208)
0925 14:57:35.404225 (+ 3805us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:35.404235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:35.404287 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:35.404574 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:35.404575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:35.404577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:35.404605 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.404606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.405878 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.405879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.406901 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.406905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.407902 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.407903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.407960 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.407964 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:35.408012 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:57:35.408049 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3791,"thread_start_us":67,"threads_started":1}
I20260925 14:57:35.408574 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.400496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.9.141.66:39589)
0925 14:57:35.404053 (+ 3557us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.404058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.404349 (+ 291us) server_negotiation.cc:407] Connection header received
0925 14:57:35.404388 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.404414 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.404429 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.404497 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:35.404672 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.405795 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.407004 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.407725 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.407808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.407867 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.408435 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.408485 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3523,"thread_start_us":53,"threads_started":1}
I20260925 14:57:35.409461 22798 tablet_service.cc:1467] Processing CreateTablet for tablet fbb19222a1714f62906c3ca15879287a (DEFAULT_TABLE table=test-table [id=8d4a5877e6b74c9cbf3d548d36553346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:35.409801 22798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb19222a1714f62906c3ca15879287a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:35.409878 23031 tablet_bootstrap.cc:492] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Bootstrap starting.
I20260925 14:57:35.410507 23031 tablet_bootstrap.cc:654] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:35.410794 23031 log.cc:824] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:35.411521 23031 tablet_bootstrap.cc:492] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: No bootstrap required, opened a new log
I20260925 14:57:35.411579 23031 ts_tablet_manager.cc:1397] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:57:35.412786 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.399916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45373 (local address 127.0.0.1:49836)
0925 14:57:35.404117 (+ 4201us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:35.404128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:35.404191 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:35.408716 (+ 4525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:35.408717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:35.408719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:35.408760 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.408761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.410271 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.410272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.411471 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.411474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.412546 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.412547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.412611 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.412614 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:35.412661 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:57:35.412705 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4184,"thread_start_us":76,"threads_started":1}
I20260925 14:57:35.413059 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.400170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.9.141.65:45373)
0925 14:57:35.408477 (+ 8307us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.408482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.408494 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:57:35.408530 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.408532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.408547 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.408630 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:35.408831 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.410155 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.411603 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.412330 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.412460 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.412522 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.412943 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.412983 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8267,"thread_start_us":82,"threads_started":1}
I20260925 14:57:35.414040 23031 raft_consensus.cc:348] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.414193 23031 raft_consensus.cc:374] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:35.414817 22664 tablet_service.cc:1467] Processing CreateTablet for tablet fbb19222a1714f62906c3ca15879287a (DEFAULT_TABLE table=test-table [id=8d4a5877e6b74c9cbf3d548d36553346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:35.415117 22664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbb19222a1714f62906c3ca15879287a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:35.418042 23034 tablet_bootstrap.cc:492] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: Bootstrap starting.
I20260925 14:57:35.414235 23031 raft_consensus.cc:729] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d00bb91928642aba55224edc9f205e1, State: Initialized, Role: FOLLOWER
I20260925 14:57:35.418637 23031 consensus_queue.cc:260] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.418895 23034 tablet_bootstrap.cc:654] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:35.418941 23031 ts_tablet_manager.cc:1428] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260925 14:57:35.419066 22998 heartbeater.cc:503] Master 127.9.141.126:41499 was elected leader, sending a full tablet report...
I20260925 14:57:35.419317 23034 log.cc:824] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:35.420528 23033 tablet_bootstrap.cc:492] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Bootstrap starting.
I20260925 14:57:35.420778 23034 tablet_bootstrap.cc:492] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: No bootstrap required, opened a new log
I20260925 14:57:35.420897 23034 ts_tablet_manager.cc:1397] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:57:35.421162 23033 tablet_bootstrap.cc:654] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:35.421486 23033 log.cc:824] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:35.422931 23034 raft_consensus.cc:348] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.423080 23034 raft_consensus.cc:374] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:35.423121 23034 raft_consensus.cc:729] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b90284652c24819843580bab1b2e65a, State: Initialized, Role: FOLLOWER
I20260925 14:57:35.423240 23034 consensus_queue.cc:260] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.423508 23034 ts_tablet_manager.cc:1428] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:57:35.423525 22730 heartbeater.cc:503] Master 127.9.141.126:41499 was elected leader, sending a full tablet report...
I20260925 14:57:35.424906 23033 tablet_bootstrap.cc:492] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: No bootstrap required, opened a new log
I20260925 14:57:35.424989 23033 ts_tablet_manager.cc:1397] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:57:35.426880 23033 raft_consensus.cc:348] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.427022 23033 raft_consensus.cc:374] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:35.427064 23033 raft_consensus.cc:729] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b30788779fa4f7098359a179eea60d3, State: Initialized, Role: FOLLOWER
I20260925 14:57:35.427176 23033 consensus_queue.cc:260] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.427431 23033 ts_tablet_manager.cc:1428] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:57:35.427460 22864 heartbeater.cc:503] Master 127.9.141.126:41499 was elected leader, sending a full tablet report...
W20260925 14:57:35.440558 22731 tablet.cc:2369] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:57:35 dist-test-slave-1sv7 krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for HTTP/127.9.141.84@KRBTEST.COM
W20260925 14:57:35.611117 22999 tablet.cc:2369] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:35.634075 23039 raft_consensus.cc:482] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:35.634235 23039 raft_consensus.cc:504] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.634672 23039 leader_election.cc:288] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d00bb91928642aba55224edc9f205e1 (127.9.141.67:39655), 8b90284652c24819843580bab1b2e65a (127.9.141.65:45373)
I20260925 14:57:35.640475 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.635380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:53869 (local address 127.9.141.67:39655)
0925 14:57:35.635477 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.635480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.636342 (+ 862us) server_negotiation.cc:407] Connection header received
0925 14:57:35.636503 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.636505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.636516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.636548 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:35.636811 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.637648 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.639345 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.639990 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.640047 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.640057 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.640311 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.640360 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 14:57:35.644562 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.635429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:45273 (local address 127.9.141.65:45373)
0925 14:57:35.635518 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.635520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.640585 (+ 5065us) server_negotiation.cc:407] Connection header received
0925 14:57:35.640620 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.640621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.640635 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.640689 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:35.640972 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.641777 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.643427 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.644093 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.644148 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.644157 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.644440 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.644487 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 14:57:35.644845 23043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.635156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45373 (local address 127.9.141.66:45273)
0925 14:57:35.640447 (+ 5291us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:35.640460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:35.640524 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:35.640797 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:35.640798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:35.640800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:35.640867 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.640869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.642040 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.642041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.643313 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.643316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.644223 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.644224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.644315 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.644319 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:35.644693 (+ 374us) client_negotiation.cc:240] Negotiation successful
0925 14:57:35.644741 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5271,"thread_start_us":79,"threads_started":1}
I20260925 14:57:35.645216 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.634926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39655 (local address 127.9.141.66:53869)
0925 14:57:35.636259 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:35.636275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:35.636412 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:35.636624 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:35.636625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:35.636626 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:35.636714 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.636716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.637775 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.637776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.639225 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:35.639229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.640143 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.640144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:35.640221 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.640226 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:57:35.645092 (+ 4866us) client_negotiation.cc:240] Negotiation successful
0925 14:57:35.645137 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1312,"thread_start_us":101,"threads_started":1}
I20260925 14:57:35.645210 22683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb19222a1714f62906c3ca15879287a" candidate_uuid: "3b30788779fa4f7098359a179eea60d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b90284652c24819843580bab1b2e65a" is_pre_election: true
I20260925 14:57:35.645398 22683 raft_consensus.cc:2454] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b30788779fa4f7098359a179eea60d3 in term 0.
I20260925 14:57:35.645550 22952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb19222a1714f62906c3ca15879287a" candidate_uuid: "3b30788779fa4f7098359a179eea60d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d00bb91928642aba55224edc9f205e1" is_pre_election: true
I20260925 14:57:35.645720 22952 raft_consensus.cc:2454] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b30788779fa4f7098359a179eea60d3 in term 0.
I20260925 14:57:35.645913 22753 leader_election.cc:302] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b30788779fa4f7098359a179eea60d3, 8b90284652c24819843580bab1b2e65a; no voters:
I20260925 14:57:35.646050 23039 raft_consensus.cc:2781] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:35.646095 23039 raft_consensus.cc:482] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 14:57:35.646104 22865 tablet.cc:2369] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:35.646147 23039 raft_consensus.cc:3037] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:35.647110 23039 raft_consensus.cc:504] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.647255 23039 leader_election.cc:288] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [CANDIDATE]: Term 1 election: Requested vote from peers 7d00bb91928642aba55224edc9f205e1 (127.9.141.67:39655), 8b90284652c24819843580bab1b2e65a (127.9.141.65:45373)
I20260925 14:57:35.647724 22952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb19222a1714f62906c3ca15879287a" candidate_uuid: "3b30788779fa4f7098359a179eea60d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d00bb91928642aba55224edc9f205e1"
I20260925 14:57:35.647823 22952 raft_consensus.cc:3037] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:35.648753 22952 raft_consensus.cc:2454] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b30788779fa4f7098359a179eea60d3 in term 1.
I20260925 14:57:35.648964 22754 leader_election.cc:302] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b30788779fa4f7098359a179eea60d3, 7d00bb91928642aba55224edc9f205e1; no voters:
I20260925 14:57:35.649097 22683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbb19222a1714f62906c3ca15879287a" candidate_uuid: "3b30788779fa4f7098359a179eea60d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b90284652c24819843580bab1b2e65a"
I20260925 14:57:35.649187 22683 raft_consensus.cc:3037] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:35.650084 22683 raft_consensus.cc:2454] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b30788779fa4f7098359a179eea60d3 in term 1.
I20260925 14:57:35.650331 23039 raft_consensus.cc:2781] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:35.650502 23039 raft_consensus.cc:686] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 1 LEADER]: Becoming Leader. State: Replica: 3b30788779fa4f7098359a179eea60d3, State: Running, Role: LEADER
I20260925 14:57:35.650600 23039 consensus_queue.cc:237] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } }
I20260925 14:57:35.652846 22532 catalog_manager.cc:5495] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b30788779fa4f7098359a179eea60d3 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "3b30788779fa4f7098359a179eea60d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b30788779fa4f7098359a179eea60d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39589 } health_report { overall_health: HEALTHY } } }
I20260925 14:57:35.704432 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:35.701002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.9.141.66:39589)
0925 14:57:35.701046 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:35.701048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:35.701056 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:35.701102 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:35.701103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:35.701126 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:35.701200 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:35.701420 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.702169 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.703288 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:35.703588 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:35.703657 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:35.703862 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:35.703996 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:35.704071 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:35.704302 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 14:57:35.704323 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:35.706840 22683 raft_consensus.cc:1261] T fbb19222a1714f62906c3ca15879287a P 8b90284652c24819843580bab1b2e65a [term 1 FOLLOWER]: Refusing update from remote peer 3b30788779fa4f7098359a179eea60d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:35.707175 22952 raft_consensus.cc:1261] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 1 FOLLOWER]: Refusing update from remote peer 3b30788779fa4f7098359a179eea60d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:35.707485 23039 consensus_queue.cc:1035] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d00bb91928642aba55224edc9f205e1" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:35.707830 23039 consensus_queue.cc:1035] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b90284652c24819843580bab1b2e65a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:35.712205 23047 mvcc.cc:205] Tried to move back new op lower bound from 7333266455370739712 to 7333266455145492480. Current Snapshot: MvccSnapshot[applied={T|T < 7333266455370739712}]
I20260925 14:57:35.713258 23050 mvcc.cc:205] Tried to move back new op lower bound from 7333266455370739712 to 7333266455145492480. Current Snapshot: MvccSnapshot[applied={T|T < 7333266455370739712}]
I20260925 14:57:35.715683 23048 mvcc.cc:205] Tried to move back new op lower bound from 7333266455370739712 to 7333266455145492480. Current Snapshot: MvccSnapshot[applied={T|T < 7333266455370739712}]
I20260925 14:57:35.718564 22532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:35.718662 22532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:35.719321 22532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 444d771a1ae24584ae8cf91e8990fd01: Sending DeleteTablet for 3 replicas of tablet fbb19222a1714f62906c3ca15879287a
I20260925 14:57:35.719754 22914 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb19222a1714f62906c3ca15879287a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:35 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:33686
I20260925 14:57:35.719954 22664 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb19222a1714f62906c3ca15879287a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:35 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:49836
I20260925 14:57:35.720287 22798 tablet_service.cc:1514] Processing DeleteTablet for tablet fbb19222a1714f62906c3ca15879287a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:35 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:58208
I20260925 14:57:35.720800 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22570
I20260925 14:57:35.723417 23058 tablet_replica.cc:330] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: stopping tablet replica
I20260925 14:57:35.723548 23058 raft_consensus.cc:2229] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:35.723695 23058 raft_consensus.cc:2258] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:35.723863 23056 tablet_replica.cc:330] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: stopping tablet replica
I20260925 14:57:35.723965 23056 raft_consensus.cc:2229] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:35.724072 23056 raft_consensus.cc:2258] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 14:57:35.727092 22518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:45373 (error 108)
I20260925 14:57:35.727914 23058 ts_tablet_manager.cc:1905] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:35.729703 23058 ts_tablet_manager.cc:1918] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:35.729836 23058 log.cc:1196] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/fbb19222a1714f62906c3ca15879287a
I20260925 14:57:35.730176 23058 ts_tablet_manager.cc:1939] T fbb19222a1714f62906c3ca15879287a P 3b30788779fa4f7098359a179eea60d3: Deleting consensus metadata
I20260925 14:57:35.730634 22516 catalog_manager.cc:4841] TS 3b30788779fa4f7098359a179eea60d3 (127.9.141.66:39589): tablet fbb19222a1714f62906c3ca15879287a (table test-table [id=8d4a5877e6b74c9cbf3d548d36553346]) successfully deleted
I20260925 14:57:35.732687 23056 ts_tablet_manager.cc:1905] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:35.734304 23056 ts_tablet_manager.cc:1918] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:35.734445 23056 log.cc:1196] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/fbb19222a1714f62906c3ca15879287a
I20260925 14:57:35.734764 23056 ts_tablet_manager.cc:1939] T fbb19222a1714f62906c3ca15879287a P 7d00bb91928642aba55224edc9f205e1: Deleting consensus metadata
I20260925 14:57:35.735244 22519 catalog_manager.cc:4841] TS 7d00bb91928642aba55224edc9f205e1 (127.9.141.67:39655): tablet fbb19222a1714f62906c3ca15879287a (table test-table [id=8d4a5877e6b74c9cbf3d548d36553346]) successfully deleted
W20260925 14:57:35.735399 23025 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:35.727525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:45373 (local address 127.0.0.1:49852)
0925 14:57:35.727582 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:35.727596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:35.727638 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:35.735354 (+ 7716us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:45373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:57:35.735512 22518 catalog_manager.cc:4584] TS 8b90284652c24819843580bab1b2e65a (127.9.141.65:45373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbb19222a1714f62906c3ca15879287a: Network error: Client connection negotiation failed: client connection to 127.9.141.65:45373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:57:35.735967 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22734
I20260925 14:57:35.742022 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22867
I20260925 14:57:35.749704 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 22499
I20260925 14:57:35.756739 9781 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:57:36.101173 9781 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:57:36.101298 9781 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:57:36.445714 9781 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:57:36.445 UTC [21566] LOG: received smart shutdown request
2026-09-25 14:57:36.449 UTC [21566] LOG: background worker "logical replication launcher" (PID 21575) exited with exit code 1
2026-09-25 14:57:36.450 UTC [21570] LOG: shutting down
2026-09-25 14:57:36.630 UTC [21566] LOG: database system is shut down
2026-09-25T14:57:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49247 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:36 dist-test-slave-1sv7 krb5kdc[23068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:36 dist-test-slave-1sv7 krb5kdc[23068](info): set up 2 sockets
Sep 25 14:57:36 dist-test-slave-1sv7 krb5kdc[23068](info): commencing operation
krb5kdc: starting...
W20260925 14:57:37.788286 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:57:37 dist-test-slave-1sv7 krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:57:37.916275 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44401
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:44401,127.9.141.125:40765,127.9.141.124:39507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:37.997174 23081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:37.997423 23081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:37.997462 23081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:37.998596 23081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:37.998665 23081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:37.998700 23081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:37.998749 23081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:37.998777 23081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:38.000280 23081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:44401,127.9.141.125:40765,127.9.141.124:39507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:38.000594 23081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:38.000831 23081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:38.005671 23086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:38.005666 23087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:38.005889 23081 server_base.cc:1034] running on GCE node
W20260925 14:57:38.005961 23089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:38.006242 23081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:38.006491 23081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:38.007670 23081 hybrid_clock.cc:648] HybridClock initialized: now 1790348258007662 us; error 38 us; skew 500 ppm
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:38.015530 23081 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:38.016098 23081 webserver.cc:460] Webserver started at http://127.9.141.126:36809/ using document root <none> and password file <none>
I20260925 14:57:38.016345 23081 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:38.016456 23081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:38.016620 23081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:38.017719 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "c77894ff7ddb40968d29ecdf00211a75"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "b06fda746af92951105acce2ad0950f5"
server_key_iv: "c979f5f9c04cd14adc39cb976dc3d94d"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.018105 23081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "c77894ff7ddb40968d29ecdf00211a75"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "b06fda746af92951105acce2ad0950f5"
server_key_iv: "c979f5f9c04cd14adc39cb976dc3d94d"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.019440 23081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:38.020212 23096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:38.020483 23081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:38.020568 23081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "c77894ff7ddb40968d29ecdf00211a75"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "b06fda746af92951105acce2ad0950f5"
server_key_iv: "c979f5f9c04cd14adc39cb976dc3d94d"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.020623 23081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:38.029950 23081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:38.032210 23081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:38.032341 23081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:38.036347 23081 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:44401
I20260925 14:57:38.036484 23148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:44401 every 8 connection(s)
I20260925 14:57:38.036819 23081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:38.037683 23149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:38.039287 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:38.041320 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23081
I20260925 14:57:38.041415 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:57:38.041633 9781 external_mini_cluster.cc:1428] Setting key 9a45f05e40d3037b3a70e6c887237adf
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.125@KRBTEST.COM; defaulting to no policy
I20260925 14:57:38.051175 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.039998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44401 (local address 127.0.0.1:38936)
0925 14:57:38.040263 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.040293 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.040806 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.041121 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.041123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.041391 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.041605 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.041611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.043149 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.043151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.044302 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.044305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.045142 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.045142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.045203 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.045805 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.045817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.047460 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.049590 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.049596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.049598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.049830 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.050107 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.050108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.050108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.050187 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.050667 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.050673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.050731 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.050840 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.050973 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"spinlock_wait_cycles":8576,"thread_start_us":105,"threads_started":1}
I20260925 14:57:38.051179 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.040006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.9.141.126:44401)
0925 14:57:38.040206 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.040216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.040431 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:57:38.040968 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.040970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.040987 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.041073 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.041710 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.043030 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.044436 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.045044 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.045156 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.047561 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.047571 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.047571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.047580 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.049480 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.049924 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.049928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.049928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.050031 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.050256 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.050258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.050258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.050590 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.050684 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.050883 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.051033 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260925 14:57:38.052126 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:38.052598 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.052476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56140)
0925 14:57:38.052528 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.052577 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:57:38.052731 23100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
I20260925 14:57:38.053133 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.053035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56150)
0925 14:57:38.053085 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.053109 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.125@KRBTEST.COM" created.
W20260925 14:57:38.053261 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:40765: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab.
Entry for principal kudu/127.9.141.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab.
I20260925 14:57:38.067939 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:40765
--webserver_interface=127.9.141.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab
--principal=kudu/127.9.141.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:44401,127.9.141.125:40765,127.9.141.124:39507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:38.102517 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } attempt: 1
I20260925 14:57:38.103073 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.102926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56154)
0925 14:57:38.103002 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.103041 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 14:57:38.103531 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.103438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56156)
0925 14:57:38.103487 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.103514 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:57:38.103652 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:40765: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
W20260925 14:57:38.149176 23156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:38.149391 23156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:38.149413 23156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:38.150602 23156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:38.150640 23156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:38.150655 23156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:38.150668 23156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:38.150681 23156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:38.152241 23156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:44401,127.9.141.125:40765,127.9.141.124:39507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab
--principal=kudu/127.9.141.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.141.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:38.152498 23156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:38.152683 23156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:38.157480 23162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:38.157473 23161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:38.157472 23164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:38.157891 23156 server_base.cc:1034] running on GCE node
I20260925 14:57:38.158149 23156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:38.158442 23156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:38.159611 23156 hybrid_clock.cc:648] HybridClock initialized: now 1790348258159593 us; error 40 us; skew 500 ppm
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:38.162230 23156 init.cc:377] Logged in from keytab as kudu/127.9.141.125@KRBTEST.COM (short username kudu)
I20260925 14:57:38.162665 23156 webserver.cc:460] Webserver started at http://127.9.141.125:39785/ using document root <none> and password file <none>
I20260925 14:57:38.162880 23156 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:38.162926 23156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:38.163074 23156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:38.164119 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data/instance:
uuid: "b84dba995f9f4dc2b93a49a342d76487"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "3980d723885746e595d57cbb9b2c0d10"
server_key_iv: "29c06bfa37fff14c33a8da594be8b5c1"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.164505 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal/instance:
uuid: "b84dba995f9f4dc2b93a49a342d76487"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "3980d723885746e595d57cbb9b2c0d10"
server_key_iv: "29c06bfa37fff14c33a8da594be8b5c1"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.165822 23156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:38.166543 23171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:38.166752 23156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:38.166878 23156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal
uuid: "b84dba995f9f4dc2b93a49a342d76487"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "3980d723885746e595d57cbb9b2c0d10"
server_key_iv: "29c06bfa37fff14c33a8da594be8b5c1"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.166963 23156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:38.171811 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } attempt: 2
I20260925 14:57:38.172209 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.172072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56170)
0925 14:57:38.172129 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.172166 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:57:38.172597 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.172509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56184)
0925 14:57:38.172556 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.172581 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:57:38.172685 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:40765: Network error: Client connection negotiation failed: client connection to 127.9.141.125:40765: connect: Connection refused (error 111)
I20260925 14:57:38.181576 23156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:38.183598 23156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:38.183756 23156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:38.187668 23156 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.125:40765
I20260925 14:57:38.187734 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.125:40765 every 8 connection(s)
I20260925 14:57:38.188063 23156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
I20260925 14:57:38.188956 23224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:38.190490 23224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:38.192833 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23156
I20260925 14:57:38.192960 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-1/wal/instance
I20260925 14:57:38.193141 9781 external_mini_cluster.cc:1428] Setting key 13aafd09a27d6ccfbfff5691b106273a
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:38.200031 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.191405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.9.141.126:44401)
0925 14:57:38.191464 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.191467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.191477 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:38.191987 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.191989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.192002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.192093 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.192773 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.193407 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.194383 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.194606 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.194686 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.197198 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.197199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.197199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.197205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.198591 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.199067 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.199068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.199068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.199121 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.199410 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.199410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.199411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.199511 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.199561 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.199899 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.199968 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:57:38.200290 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.191151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44401 (local address 127.0.0.1:38952)
0925 14:57:38.191321 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.191364 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.191873 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.192169 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.192174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.192485 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.192676 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.192687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.193497 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.193498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.194273 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.194275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.194697 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.194698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.194820 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.195331 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.195341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.197097 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.198678 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.198686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.198688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.198987 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.199199 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.199200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.199201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.199340 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.199581 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.199586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.199745 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.199866 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.200032 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.9.141.124@KRBTEST.COM; defaulting to no policy
I20260925 14:57:38.200929 23224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.124@KRBTEST.COM" created.
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:57:38.208840 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.201217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56190)
0925 14:57:38.201270 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.201295 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.201344 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.201578 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.201579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.201720 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.201767 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.201768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.202732 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.202733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.203219 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.203220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.203811 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.203812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.203871 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.204206 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.204211 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.205707 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.207745 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.207745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.207746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.207901 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.208152 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.208153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.208153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.208212 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.208608 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.208608 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.208644 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.208701 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.208779 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:57:38.209105 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.201289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.9.141.125:40765)
0925 14:57:38.201438 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.201443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.201462 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:57:38.201486 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.201488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.201498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.201578 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.201865 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.202626 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.203318 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.203723 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.203812 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.205812 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.205819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.205819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.205825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.207666 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.207986 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.207989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.207989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.208053 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.208290 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.208291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.208291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.208530 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.208625 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.208738 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.208868 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260925 14:57:38.209753 23224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:38.210108 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.209976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45092)
0925 14:57:38.210029 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.210082 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:57:38.210264 23172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
I20260925 14:57:38.210758 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.210646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45102)
0925 14:57:38.210698 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.210731 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:57:38.210996 23224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:39507: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab.
Entry for principal kudu/127.9.141.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab.
I20260925 14:57:38.219573 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:39507
--webserver_interface=127.9.141.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab
--principal=kudu/127.9.141.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:44401,127.9.141.125:40765,127.9.141.124:39507
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:38.260257 23224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } attempt: 1
I20260925 14:57:38.260847 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.260693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45114)
0925 14:57:38.260766 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.260814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 14:57:38.261258 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.261138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45128)
0925 14:57:38.261195 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.261228 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:57:38.261391 23224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:39507: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
I20260925 14:57:38.263866 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } attempt: 3
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:57:38.271065 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.264246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.9.141.125:40765)
0925 14:57:38.264333 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.264335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.264347 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:57:38.264405 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.264406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.264417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.264475 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.264842 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.265356 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.266107 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.266335 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.266427 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.268290 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.268292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.268292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.268300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.269891 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.270235 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.270237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.270237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.270303 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.270505 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.270506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.270506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.270643 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.270717 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.270887 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.270970 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:57:38.271121 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.264187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56198)
0925 14:57:38.264227 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.264239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.264320 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.264481 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.264482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.264654 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.264717 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.264720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.265446 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.265447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.266017 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.266019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.266429 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.266430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.266508 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.266984 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.266989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.268192 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.269971 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.269972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.269973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.270152 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.270370 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.270371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.270372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.270438 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.270719 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.270721 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.270768 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.270861 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.271022 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 14:57:38.271461 23149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:38.271875 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.271730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45142)
0925 14:57:38.271781 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.271854 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:57:38.272297 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.272212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45152)
0925 14:57:38.272257 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.272279 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:57:38.272414 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:39507: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
W20260925 14:57:38.302340 23231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:38.302598 23231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:38.302640 23231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 14:57:38.303557 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } attempt: 1
W20260925 14:57:38.304010 23231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260925 14:57:38.304042 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.303904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45160)
0925 14:57:38.303974 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.304013 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 14:57:38.304075 23231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:38.304106 23231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:38.304138 23231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:38.304157 23231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:38.304409 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.304272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45168)
0925 14:57:38.304324 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.304348 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:57:38.304492 23149 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:39507: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
I20260925 14:57:38.305940 23231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:44401,127.9.141.125:40765,127.9.141.124:39507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab
--principal=kudu/127.9.141.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:39507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.141.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:38.306216 23231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:38.306473 23231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:38.311880 23239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:38.311897 23237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:38.311947 23231 server_base.cc:1034] running on GCE node
W20260925 14:57:38.311895 23236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:38.312325 23231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:38.312695 23231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:38.313874 23231 hybrid_clock.cc:648] HybridClock initialized: now 1790348258313857 us; error 37 us; skew 500 ppm
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:38.316483 23231 init.cc:377] Logged in from keytab as kudu/127.9.141.124@KRBTEST.COM (short username kudu)
I20260925 14:57:38.316956 23231 webserver.cc:460] Webserver started at http://127.9.141.124:36613/ using document root <none> and password file <none>
I20260925 14:57:38.317185 23231 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:38.317252 23231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:38.317394 23231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:38.318440 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data/instance:
uuid: "fa5fd087994f4d1b8b3125fe8c7e7933"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "ac6fff9b2e7f54da01f0860e3ed046d9"
server_key_iv: "c15609909422f03b57531a33b8e47572"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.318812 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal/instance:
uuid: "fa5fd087994f4d1b8b3125fe8c7e7933"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "ac6fff9b2e7f54da01f0860e3ed046d9"
server_key_iv: "c15609909422f03b57531a33b8e47572"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.320128 23231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:57:38.320964 23246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:38.321166 23231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:38.321280 23231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal
uuid: "fa5fd087994f4d1b8b3125fe8c7e7933"
format_stamp: "Formatted at 2026-09-25 14:57:38 on dist-test-slave-1sv7"
server_key: "ac6fff9b2e7f54da01f0860e3ed046d9"
server_key_iv: "c15609909422f03b57531a33b8e47572"
server_key_version: "encryptionkey@0"
I20260925 14:57:38.321372 23231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:38.329548 23224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } attempt: 2
I20260925 14:57:38.329990 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.329844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45182)
0925 14:57:38.329909 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.329953 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:57:38.330380 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.330305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45196)
0925 14:57:38.330342 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.330365 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 14:57:38.330521 23224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:39507: Network error: Client connection negotiation failed: client connection to 127.9.141.124:39507: connect: Connection refused (error 111)
I20260925 14:57:38.336606 23231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:38.338409 23231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:38.338567 23231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:38.342681 23231 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.124:39507
I20260925 14:57:38.342715 23298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.124:39507 every 8 connection(s)
I20260925 14:57:38.343092 23231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
I20260925 14:57:38.343932 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:38.344333 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23231
I20260925 14:57:38.344461 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348105363960-9781-0/minicluster-data/master-2/wal/instance
I20260925 14:57:38.344635 9781 external_mini_cluster.cc:1428] Setting key 8645d5b104557ef02bdaac2414fa6cf3
I20260925 14:57:38.346079 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:57:38.355300 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.346944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.9.141.126:44401)
0925 14:57:38.347014 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.347017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.347189 (+ 172us) server_negotiation.cc:407] Connection header received
0925 14:57:38.347263 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.347264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.347276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.347334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.347840 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.348311 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.349208 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.349401 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.349483 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.351660 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.351662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.351662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.351668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.353811 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.354285 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.354287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.354287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.354351 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.354586 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.354587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.354587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.354735 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.354798 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.355122 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.355207 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:57:38.355527 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.346861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44401 (local address 127.0.0.1:38964)
0925 14:57:38.347079 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.347116 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.347209 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.347353 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.347356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.347644 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.347743 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.347748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.348423 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.348425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.349103 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.349105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.349495 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.349496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.349573 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.350104 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.350114 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.351559 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.353906 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.353913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.353915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.354211 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.354421 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.354422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.354422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.354518 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.354802 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.354806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.354980 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.355096 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.355255 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260925 14:57:38.355818 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.345155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.9.141.124:39507)
0925 14:57:38.345368 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.345378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.345415 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:57:38.345981 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.345990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.346052 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.346207 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.347055 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.347961 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.348808 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.349255 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.349384 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.352267 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.352276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.352276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.352283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.354369 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.354750 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.354752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.354753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.354822 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.354999 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.354999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.355000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.355157 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.355313 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.355614 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.355696 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260925 14:57:38.356091 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:38.364423 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.356449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.9.141.125:40765)
0925 14:57:38.356492 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.356494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.356501 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:38.356524 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.356525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.356535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.356585 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.356842 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.357676 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.358716 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.359055 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.359119 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.361156 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.361158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.361158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.361166 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.362793 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.363643 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.363645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.363645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.363706 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.363896 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.363897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.363898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.364016 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.364086 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.364245 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.364314 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:57:38.364445 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.356336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:40765 (local address 127.0.0.1:56208)
0925 14:57:38.356408 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.356419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.356459 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.356589 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.356590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.356724 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.356763 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.356765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.357763 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.357764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.358612 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.358614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.359151 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.359152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.359250 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.359747 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.359753 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.361069 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.363420 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.363421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.363421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.363569 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.363778 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.363779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.363780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.363837 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.364078 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.364079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.364121 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.364217 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.364309 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:57:38.365027 23299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:57:38.365358 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.356753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.9.141.126:44401)
0925 14:57:38.356791 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.356792 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.356848 (+ 56us) server_negotiation.cc:407] Connection header received
0925 14:57:38.356913 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.356914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.356921 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.356956 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.357643 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.358270 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.359182 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.359465 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.359507 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.361977 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.361979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.361979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.361985 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.363343 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.363740 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.363741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.363742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.363820 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.364034 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.364035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.364036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.364169 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.365087 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.365136 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.365228 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:57:38.372167 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.365341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45212)
0925 14:57:38.365384 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.365394 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.365434 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.365608 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.365608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.365795 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.365830 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.365832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.366604 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.366605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.367241 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.367244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.367590 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.367591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.367650 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.367997 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.368003 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.369326 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.371143 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.371144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.371144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.371355 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.371554 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.371554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.371555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.371607 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.371846 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.371847 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.371882 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.371962 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.372057 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:57:38.372220 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.365404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.9.141.124:39507)
0925 14:57:38.365447 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.365449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.365458 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:38.365504 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.365505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.365515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.365569 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.365911 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.366516 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.367323 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.367516 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.367564 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.369417 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.369417 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.369418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.369424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.370850 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.371439 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.371440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.371440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.371491 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.371673 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.371673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.371674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.371774 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.371842 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.371983 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.372108 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":22}
I20260925 14:57:38.373898 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933: Bootstrap starting.
I20260925 14:57:38.374583 23299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:38.374887 23299 log.cc:824] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:38.375890 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933: No bootstrap required, opened a new log
I20260925 14:57:38.376595 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.366046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.9.141.125:40765)
0925 14:57:38.366090 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.366092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.366101 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:38.366172 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.366173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.366182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.366244 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.366758 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.367164 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.368082 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.368260 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.368311 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.370938 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.370939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.370940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.370947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.375213 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.375685 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.375687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.375687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.375753 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.375947 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.375948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.375949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.376101 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.376314 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.376356 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.376485 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:38.378204 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:38.378281 23299 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.378434 23299 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:38.378465 23299 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5fd087994f4d1b8b3125fe8c7e7933, State: Initialized, Role: FOLLOWER
I20260925 14:57:38.378587 23299 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.378882 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.378940 23299 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:38.378991 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:38.379606 23149 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } attempt: 2
W20260925 14:57:38.381067 23323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:38.381125 23323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:38.381330 23231 master_runner.cc:186] Error getting master registration for 127.9.141.126:44401: OK, instance_id { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" instance_seqno: 1790348258035261 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:38.384313 23231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:57:38.388115 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.379918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.9.141.124:39507)
0925 14:57:38.380014 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.380016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.380061 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:57:38.380121 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.380122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.380134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.380219 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.381111 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.381782 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.382452 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.382654 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.382723 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.385170 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.385171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.385171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.385178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.386720 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.387150 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.387151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.387152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.387208 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.387420 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.387421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.387421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.387593 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.387666 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.387912 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.388008 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 14:57:38.389926 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.379951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.9.141.124:39507)
0925 14:57:38.381216 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.381218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.381226 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:38.381259 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.381260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.381270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.381344 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.381646 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.382519 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.383639 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.383969 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.384036 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.386281 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.386282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.386283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.386289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.388486 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.388908 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.388909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.388909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.388994 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.389251 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.389252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.389253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.389379 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.389699 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.389740 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.389817 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1243,"thread_start_us":75,"threads_started":1}
I20260925 14:57:38.390139 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.379842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45230)
0925 14:57:38.379892 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.379903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.379940 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.381361 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.381362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.381519 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.381560 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.381562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.382655 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.382656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.383526 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.383529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.384060 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.384062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.384111 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.384578 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.384584 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.386178 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.388585 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.388586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.388586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.388822 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.389098 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.389099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.389099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.389170 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.389462 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.389463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.389509 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.389559 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.390036 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:57:38.391782 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: Bootstrap starting.
I20260925 14:57:38.392432 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.380155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.9.141.126:44401)
0925 14:57:38.380191 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.380193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.380454 (+ 261us) server_negotiation.cc:407] Connection header received
0925 14:57:38.380584 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.380585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.380596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.380629 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.385076 (+ 4447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.385788 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.386899 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.387485 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.387547 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.389143 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.389144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.389144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.389151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.391009 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.391508 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.391509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.391510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.391576 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.391810 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.391812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.391812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.391946 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.392026 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.392235 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.392322 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:57:38.392575 23149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:38.392912 23149 log.cc:824] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:38.393707 23149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: No bootstrap required, opened a new log
I20260925 14:57:38.395336 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.380478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.9.141.125:40765)
0925 14:57:38.380518 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.380520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.386996 (+ 6476us) server_negotiation.cc:407] Connection header received
0925 14:57:38.387107 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.387108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.387121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.387158 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.387910 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.388732 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.390169 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.390482 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.390550 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.392007 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.392009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.392009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.392016 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.393725 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.394122 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.394123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.394124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.394182 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.394391 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.394393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.394393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.394512 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.395171 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.395207 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.395274 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:38.395723 23149 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.395864 23149 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:38.395902 23149 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77894ff7ddb40968d29ecdf00211a75, State: Initialized, Role: FOLLOWER
I20260925 14:57:38.396028 23149 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.396291 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.396345 23149 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:38.397857 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:57:38.398464 23338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:38.398573 23338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:38.399104 23081 master_runner.cc:186] Error getting master registration for 127.9.141.125:40765: OK, instance_id { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" instance_seqno: 1790348258186596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:57:38.402037 23081 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:38.421712 23224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } attempt: 3
Sep 25 14:57:38 dist-test-slave-1sv7 krb5kdc[23068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348258, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:57:38.430778 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.422011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:39507 (local address 127.0.0.1:45242)
0925 14:57:38.422065 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:38.422076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:38.422112 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:38.422280 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:38.422281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:38.422481 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:38.422556 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.422558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.423677 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.423678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.424605 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:38.424608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.425170 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.425171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:38.425234 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.425864 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:38.425870 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:38.427474 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:38.429618 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.429619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.429620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.429842 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.430067 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:38.430068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:38.430068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:38.430136 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:38.430430 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:38.430431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:38.430481 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:57:38.430557 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:57:38.430671 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:57:38.430961 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:38.422091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.9.141.124:39507)
0925 14:57:38.422138 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:38.422141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:38.422152 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:38.422179 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:38.422180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:38.422192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:38.422267 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:38.422640 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.423566 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.424683 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:38.425046 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:38.425157 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:38.427571 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:38.427573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:38.427573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:38.427581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:38.429528 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.429925 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.429926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.429926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.429993 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:38.430206 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:38.430207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:38.430207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:38.430357 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:38.430432 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:38.430621 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:57:38.430720 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:57:38.432760 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: Bootstrap starting.
I20260925 14:57:38.433622 23224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:38.433961 23224 log.cc:824] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:38.435029 23224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: No bootstrap required, opened a new log
I20260925 14:57:38.437723 23224 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.437877 23224 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:38.437915 23224 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84dba995f9f4dc2b93a49a342d76487, State: Initialized, Role: FOLLOWER
I20260925 14:57:38.438077 23224 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.438416 23224 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:38.438376 23341 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.438565 23341 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:57:38.440348 23352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:57:38.440490 23352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:57:38.442624 23156 master_runner.cc:263] No leader master found from master b84dba995f9f4dc2b93a49a342d76487
I20260925 14:57:38.442687 23156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:57:38.505893 23305 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:38.506047 23305 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.506316 23305 leader_election.cc:288] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c77894ff7ddb40968d29ecdf00211a75 (127.9.141.126:44401), b84dba995f9f4dc2b93a49a342d76487 (127.9.141.125:40765)
I20260925 14:57:38.506685 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77894ff7ddb40968d29ecdf00211a75" is_pre_election: true
I20260925 14:57:38.506852 23123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa5fd087994f4d1b8b3125fe8c7e7933 in term 0.
I20260925 14:57:38.507042 23250 leader_election.cc:302] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77894ff7ddb40968d29ecdf00211a75, fa5fd087994f4d1b8b3125fe8c7e7933; no voters:
I20260925 14:57:38.507184 23305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:38.507244 23305 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:38.507270 23305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:38.508229 23305 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.508574 23123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77894ff7ddb40968d29ecdf00211a75"
I20260925 14:57:38.508654 23123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:38.509609 23123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5fd087994f4d1b8b3125fe8c7e7933 in term 1.
I20260925 14:57:38.509781 23250 leader_election.cc:302] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77894ff7ddb40968d29ecdf00211a75, fa5fd087994f4d1b8b3125fe8c7e7933; no voters:
I20260925 14:57:38.508356 23305 leader_election.cc:288] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [CANDIDATE]: Term 1 election: Requested vote from peers c77894ff7ddb40968d29ecdf00211a75 (127.9.141.126:44401), b84dba995f9f4dc2b93a49a342d76487 (127.9.141.125:40765)
I20260925 14:57:38.510290 23305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:38.510221 23197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84dba995f9f4dc2b93a49a342d76487"
I20260925 14:57:38.510339 23305 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [term 1 LEADER]: Becoming Leader. State: Replica: fa5fd087994f4d1b8b3125fe8c7e7933, State: Running, Role: LEADER
I20260925 14:57:38.510363 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:38.510421 23305 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } }
I20260925 14:57:38.510859 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5fd087994f4d1b8b3125fe8c7e7933. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.510943 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.511224 23356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:38.511281 23197 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa5fd087994f4d1b8b3125fe8c7e7933 in term 1.
I20260925 14:57:38.511422 23356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:38.510232 23198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84dba995f9f4dc2b93a49a342d76487" is_pre_election: true
I20260925 14:57:38.511641 23198 raft_consensus.cc:2362] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fa5fd087994f4d1b8b3125fe8c7e7933 in term 1. Re-sending same reply.
I20260925 14:57:38.514387 23198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [term 1 FOLLOWER]: Refusing update from remote peer fa5fd087994f4d1b8b3125fe8c7e7933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:38.514698 23305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:38.515957 23341 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5fd087994f4d1b8b3125fe8c7e7933. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.516057 23341 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:38.516709 23123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [term 1 FOLLOWER]: Refusing update from remote peer fa5fd087994f4d1b8b3125fe8c7e7933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:38.516966 23305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:38.518151 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5fd087994f4d1b8b3125fe8c7e7933. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.518273 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:38.519224 23354 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.519321 23354 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.519630 23327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.519717 23327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:38.519975 23357 mvcc.cc:205] Tried to move back new op lower bound from 7333266466872606720 to 7333266466859253760. Current Snapshot: MvccSnapshot[applied={T|T < 7333266466872606720}]
I20260925 14:57:38.520423 23360 mvcc.cc:205] Tried to move back new op lower bound from 7333266466872606720 to 7333266466859253760. Current Snapshot: MvccSnapshot[applied={T|T < 7333266466872606720}]
I20260925 14:57:38.521292 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.521382 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:38.521750 23341 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77894ff7ddb40968d29ecdf00211a75" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44401 } } peers { permanent_uuid: "b84dba995f9f4dc2b93a49a342d76487" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 40765 } } peers { permanent_uuid: "fa5fd087994f4d1b8b3125fe8c7e7933" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 39507 } } }
I20260925 14:57:38.521842 23341 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:57:38.522469 23358 mvcc.cc:205] Tried to move back new op lower bound from 7333266466872606720 to 7333266466859253760. Current Snapshot: MvccSnapshot[applied={T|T < 7333266466872606720}]
I20260925 14:57:38.523030 23356 catalog_manager.cc:1347] Generated new cluster ID: 216adfc8a8d1447da96c9ce0922dd134
I20260925 14:57:38.523083 23356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:38.563576 23356 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:38.564224 23356 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:38.573200 23356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5fd087994f4d1b8b3125fe8c7e7933: Generated new TSK 0
I20260925 14:57:38.573402 23356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:39.399111 23338 catalog_manager.cc:1259] Loaded cluster ID: 216adfc8a8d1447da96c9ce0922dd134
I20260925 14:57:39.399221 23338 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: loading cluster ID for follower catalog manager: success
I20260925 14:57:39.400040 23338 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: acquiring CA information for follower catalog manager: success
I20260925 14:57:39.400347 23338 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c77894ff7ddb40968d29ecdf00211a75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:57:39.440953 23352 catalog_manager.cc:1259] Loaded cluster ID: 216adfc8a8d1447da96c9ce0922dd134
I20260925 14:57:39.441028 23352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: loading cluster ID for follower catalog manager: success
I20260925 14:57:39.441807 23352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: acquiring CA information for follower catalog manager: success
I20260925 14:57:39.442114 23352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b84dba995f9f4dc2b93a49a342d76487: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:57:39.632591 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23081
I20260925 14:57:39.637712 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23156
I20260925 14:57:39.642642 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23231
2026-09-25T14:57:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2901 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:39 dist-test-slave-1sv7 krb5kdc[23372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:39 dist-test-slave-1sv7 krb5kdc[23372](info): set up 2 sockets
Sep 25 14:57:39 dist-test-slave-1sv7 krb5kdc[23372](info): commencing operation
krb5kdc: starting...
W20260925 14:57:40.716964 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:57:40 dist-test-slave-1sv7 krb5kdc[23372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:57:40.844110 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39563
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:40.926591 23385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:40.926829 23385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:40.926875 23385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:40.928117 23385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:40.928189 23385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:40.928215 23385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:40.928244 23385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:40.928273 23385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:40.929765 23385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:39563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:40.930090 23385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:40.930318 23385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:57:40.935292 23385 server_base.cc:1034] running on GCE node
W20260925 14:57:40.935315 23391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:40.935470 23390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:40.935307 23393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:40.935737 23385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:40.936041 23385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:40.937228 23385 hybrid_clock.cc:648] HybridClock initialized: now 1790348260937211 us; error 47 us; skew 500 ppm
Sep 25 14:57:40 dist-test-slave-1sv7 krb5kdc[23372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348260, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:40.939905 23385 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:40.940527 23385 webserver.cc:460] Webserver started at http://127.9.141.126:43561/ using document root <none> and password file <none>
I20260925 14:57:40.940762 23385 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:40.940825 23385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:40.940961 23385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:40.942068 23385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "eb98039157c54700a19987edda3f1edf"
format_stamp: "Formatted at 2026-09-25 14:57:40 on dist-test-slave-1sv7"
server_key: "e519287400de0e0a13c67c8399cce752"
server_key_iv: "66f83c0f0a35a9d4eb86c178987c6575"
server_key_version: "encryptionkey@0"
I20260925 14:57:40.942438 23385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "eb98039157c54700a19987edda3f1edf"
format_stamp: "Formatted at 2026-09-25 14:57:40 on dist-test-slave-1sv7"
server_key: "e519287400de0e0a13c67c8399cce752"
server_key_iv: "66f83c0f0a35a9d4eb86c178987c6575"
server_key_version: "encryptionkey@0"
I20260925 14:57:40.943809 23385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:40.944628 23400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:40.944787 23385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:40.944883 23385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "eb98039157c54700a19987edda3f1edf"
format_stamp: "Formatted at 2026-09-25 14:57:40 on dist-test-slave-1sv7"
server_key: "e519287400de0e0a13c67c8399cce752"
server_key_iv: "66f83c0f0a35a9d4eb86c178987c6575"
server_key_version: "encryptionkey@0"
I20260925 14:57:40.945019 23385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:40.961684 23385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:40.963760 23385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:40.963977 23385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:40.968070 23385 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:39563
I20260925 14:57:40.968117 23452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:39563 every 8 connection(s)
I20260925 14:57:40.968515 23385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:40.969120 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23385
I20260925 14:57:40.969223 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:57:40.969453 9781 external_mini_cluster.cc:1428] Setting key cf33025e2af4242039ec56a9b3e6cd78
I20260925 14:57:40.969484 23453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:40.972236 23453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf: Bootstrap starting.
I20260925 14:57:40.972877 23453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:40.973156 23453 log.cc:824] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:40.973851 23453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf: No bootstrap required, opened a new log
Sep 25 14:57:40 dist-test-slave-1sv7 krb5kdc[23372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:40.975662 23453 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb98039157c54700a19987edda3f1edf" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39563 } }
I20260925 14:57:40.975858 23453 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:40.975900 23453 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb98039157c54700a19987edda3f1edf, State: Initialized, Role: FOLLOWER
I20260925 14:57:40.976050 23453 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb98039157c54700a19987edda3f1edf" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39563 } }
I20260925 14:57:40.976130 23453 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:40.976153 23453 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:40.976218 23453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:40.976976 23453 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb98039157c54700a19987edda3f1edf" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39563 } }
I20260925 14:57:40.977082 23453 leader_election.cc:302] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb98039157c54700a19987edda3f1edf; no voters:
I20260925 14:57:40.977250 23453 leader_election.cc:288] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:40.977320 23458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:40.977483 23453 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:40.977555 23458 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [term 1 LEADER]: Becoming Leader. State: Replica: eb98039157c54700a19987edda3f1edf, State: Running, Role: LEADER
I20260925 14:57:40.977675 23458 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb98039157c54700a19987edda3f1edf" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39563 } }
I20260925 14:57:40.978005 23458 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb98039157c54700a19987edda3f1edf. Latest consensus state: current_term: 1 leader_uuid: "eb98039157c54700a19987edda3f1edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb98039157c54700a19987edda3f1edf" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39563 } } }
I20260925 14:57:40.978101 23458 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:40.978017 23459 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb98039157c54700a19987edda3f1edf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb98039157c54700a19987edda3f1edf" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39563 } } }
I20260925 14:57:40.978314 23459 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:40.978336 23467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:40.978907 23467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:40.979846 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:40.969918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.9.141.126:39563)
0925 14:57:40.970115 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:40.970125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:40.970162 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:57:40.970540 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:40.970545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:40.970589 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:40.970731 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:40.971336 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:40.972235 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:40.973039 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:40.973486 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:40.973591 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:40.975767 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:40.975771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:40.975772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:40.975783 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:40.977936 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:40.978367 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:40.978370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:40.978371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:40.978470 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:40.978710 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:40.978711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:40.978712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:40.979072 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:40.979234 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:40.979489 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 14:57:40.979627 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260925 14:57:40.980667 23467 catalog_manager.cc:1347] Generated new cluster ID: dfb156cc06144bd0a927b4555035365f
I20260925 14:57:40.980732 23467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:40.980798 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:57:40.994657 23467 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:40.995422 23467 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:41.000721 23467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb98039157c54700a19987edda3f1edf: Generated new TSK 0
I20260925 14:57:41.000907 23467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:57:41.034795 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23385
2026-09-25T14:57:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1390 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:41 dist-test-slave-1sv7 krb5kdc[23480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:41 dist-test-slave-1sv7 krb5kdc[23480](info): set up 2 sockets
Sep 25 14:57:41 dist-test-slave-1sv7 krb5kdc[23480](info): commencing operation
krb5kdc: starting...
W20260925 14:57:42.072562 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:57:42.191088 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39829
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:39829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:42.270457 23493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:42.270702 23493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:42.270740 23493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:42.271900 23493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:42.271967 23493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:42.271991 23493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:42.272022 23493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:42.272051 23493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:42.273541 23493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:39829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:42.273819 23493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:42.274037 23493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:42.278928 23499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.278997 23493 server_base.cc:1034] running on GCE node
W20260925 14:57:42.278914 23501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.279006 23498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.279491 23493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:42.279738 23493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:42.280874 23493 hybrid_clock.cc:648] HybridClock initialized: now 1790348262280855 us; error 37 us; skew 500 ppm
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:42.283402 23493 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:42.283880 23493 webserver.cc:460] Webserver started at http://127.9.141.126:34031/ using document root <none> and password file <none>
I20260925 14:57:42.284111 23493 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:42.284179 23493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:42.284323 23493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:42.285389 23493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "43b5c758a22e419b8e42d4afd3961f24"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "7cc39e5b0aebd28ef5c25bfb249ce129"
server_key_iv: "b26f5a7edb3d2e93482fb4f2a47de3aa"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.285753 23493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "43b5c758a22e419b8e42d4afd3961f24"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "7cc39e5b0aebd28ef5c25bfb249ce129"
server_key_iv: "b26f5a7edb3d2e93482fb4f2a47de3aa"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.287040 23493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:42.287855 23508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.288040 23493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:42.288146 23493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "43b5c758a22e419b8e42d4afd3961f24"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "7cc39e5b0aebd28ef5c25bfb249ce129"
server_key_iv: "b26f5a7edb3d2e93482fb4f2a47de3aa"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.288228 23493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:42.297395 23493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:42.299568 23493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:42.299716 23493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:42.303753 23493 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:39829
I20260925 14:57:42.303785 23560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:39829 every 8 connection(s)
I20260925 14:57:42.304127 23493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:42.305135 23561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:42.305369 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23493
I20260925 14:57:42.305478 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:57:42.305722 9781 external_mini_cluster.cc:1428] Setting key 56e9b47120c1f8a4dfe871d10eb6cb03
I20260925 14:57:42.308284 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap starting.
I20260925 14:57:42.308889 23561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:42.309247 23561 log.cc:824] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:42.310006 23561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: No bootstrap required, opened a new log
I20260925 14:57:42.312614 23561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:42.312765 23561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:42.312811 23561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b5c758a22e419b8e42d4afd3961f24, State: Initialized, Role: FOLLOWER
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:42.312939 23561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:42.313060 23561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:42.313103 23561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:42.313153 23561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:42.313673 23561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:42.313788 23561 leader_election.cc:302] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b5c758a22e419b8e42d4afd3961f24; no voters:
I20260925 14:57:42.313956 23561 leader_election.cc:288] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:57:42.314013 23566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:42.314168 23566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 LEADER]: Becoming Leader. State: Replica: 43b5c758a22e419b8e42d4afd3961f24, State: Running, Role: LEADER
I20260925 14:57:42.314245 23561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:42.314327 23566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:42.315057 23567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b5c758a22e419b8e42d4afd3961f24. Latest consensus state: current_term: 1 leader_uuid: "43b5c758a22e419b8e42d4afd3961f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } } }
I20260925 14:57:42.315066 23568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43b5c758a22e419b8e42d4afd3961f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } } }
I20260925 14:57:42.315174 23567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:42.315198 23568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:42.315519 23576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:42.316256 23576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:42.317502 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.306348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.9.141.126:39829)
0925 14:57:42.306539 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.306547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.306574 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:57:42.306995 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.307002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.307047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.307173 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.308001 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.308951 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.309991 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.310461 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.310554 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.313460 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.313464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.313464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:42.313474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.315710 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.316159 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.316162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.316162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.316269 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.316516 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.316517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.316517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.316852 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.316989 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.317163 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.317306 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260925 14:57:42.318193 23576 catalog_manager.cc:1347] Generated new cluster ID: e40de25767e3433f89fe0cbbde430631
I20260925 14:57:42.318286 23576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:42.327749 23576 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:57:42.328401 23576 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:42.333545 23576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Generated new TSK 0
I20260925 14:57:42.333734 23576 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:57:42.365584 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39829
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:42.449509 23587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:42.449712 23587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:42.449736 23587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:42.450873 23587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:42.450917 23587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:42.450942 23587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:57:42.452769 23587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:42.453037 23587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:42.453285 23587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:42.458441 23592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.458433 23595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.458433 23593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.458464 23587 server_base.cc:1034] running on GCE node
I20260925 14:57:42.458839 23587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:42.459081 23587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:42.460243 23587 hybrid_clock.cc:648] HybridClock initialized: now 1790348262460232 us; error 31 us; skew 500 ppm
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:42.462931 23587 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:57:42.463586 23587 webserver.cc:460] Webserver started at http://127.9.141.65:46495/ using document root <none> and password file <none>
I20260925 14:57:42.463843 23587 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:42.463912 23587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:42.464061 23587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:42.465157 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "40498e7071e94bb0ace7c7ee3cc75d7f"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "57870a5dd6b7a0ded1b6e129bd8a5ae6"
server_key_iv: "684a3dee8d102f843963f0d8b91e070c"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.465507 23587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "40498e7071e94bb0ace7c7ee3cc75d7f"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "57870a5dd6b7a0ded1b6e129bd8a5ae6"
server_key_iv: "684a3dee8d102f843963f0d8b91e070c"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.466809 23587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:42.467562 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.467742 23587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:57:42.467833 23587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "40498e7071e94bb0ace7c7ee3cc75d7f"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "57870a5dd6b7a0ded1b6e129bd8a5ae6"
server_key_iv: "684a3dee8d102f843963f0d8b91e070c"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.467908 23587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:42.491691 23587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:42.493999 23587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:42.494210 23587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:42.494467 23587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:42.494796 23587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:42.494849 23587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.494891 23587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:42.494920 23587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.500916 23587 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:44033
I20260925 14:57:42.500959 23715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:44033 every 8 connection(s)
I20260925 14:57:42.501298 23587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:42.510629 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.502093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:49331 (local address 127.9.141.126:39829)
0925 14:57:42.502162 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.502166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.502437 (+ 271us) server_negotiation.cc:407] Connection header received
0925 14:57:42.502981 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.502982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.502993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.503037 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.504023 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.504613 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.505649 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.505860 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.505935 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.508133 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.508134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.508134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:42.508142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.509475 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.509852 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.509852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.509853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.509903 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.510104 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.510105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.510105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.510228 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.510280 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.510502 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.510565 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:57:42.510813 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.502094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:49331)
0925 14:57:42.502297 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:42.502349 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:42.502871 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:42.503119 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:42.503123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:42.503723 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:42.503921 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.503926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.504714 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.504716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.505542 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.505544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.505942 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.505943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.506074 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.506620 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:42.506628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:42.508046 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:42.509550 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:42.509557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:42.509559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:42.509788 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:42.509961 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:42.509961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:42.509961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:42.510053 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:42.510281 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:42.510285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:42.510397 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 14:57:42.510477 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:57:42.510595 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260925 14:57:42.511363 23716 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39829
I20260925 14:57:42.511473 23716 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:42.511490 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23587
I20260925 14:57:42.511587 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:57:42.511703 23716 heartbeater.cc:511] Master 127.9.141.126:39829 requested a full tablet report, sending...
I20260925 14:57:42.511818 9781 external_mini_cluster.cc:1428] Setting key 7dad2077fc9d8af4fb9ccb0397a070cc
I20260925 14:57:42.512310 23525 ts_manager.cc:194] Registered new tserver with Master: 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033)
I20260925 14:57:42.513037 23525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:49331
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:57:42.539903 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39829
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:42.620265 23721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:42.620534 23721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:42.620563 23721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:42.621726 23721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:42.621767 23721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:42.621794 23721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:57:42.623272 23721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:42.623504 23721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:42.623730 23721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:42.628950 23729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.629040 23721 server_base.cc:1034] running on GCE node
W20260925 14:57:42.628945 23726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.628947 23727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.629380 23721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:42.629598 23721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:42.630746 23721 hybrid_clock.cc:648] HybridClock initialized: now 1790348262630738 us; error 36 us; skew 500 ppm
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:42.633353 23721 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:57:42.633782 23721 webserver.cc:460] Webserver started at http://127.9.141.66:34997/ using document root <none> and password file <none>
I20260925 14:57:42.634003 23721 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:42.634068 23721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:42.634212 23721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:42.635293 23721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "0be67d367ec24ab58418a271e15f11cb"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "e5342947b59a5343b9ab1dd0e2747cb1"
server_key_iv: "5140847d0b94f77afa7efb10dcb00608"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.635650 23721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "0be67d367ec24ab58418a271e15f11cb"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "e5342947b59a5343b9ab1dd0e2747cb1"
server_key_iv: "5140847d0b94f77afa7efb10dcb00608"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.637171 23721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:57:42.637977 23736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.638278 23721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:42.638378 23721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "0be67d367ec24ab58418a271e15f11cb"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "e5342947b59a5343b9ab1dd0e2747cb1"
server_key_iv: "5140847d0b94f77afa7efb10dcb00608"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.638459 23721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:42.663705 23721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:42.665824 23721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:42.665988 23721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:42.666241 23721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:42.666571 23721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:42.666623 23721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.666664 23721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:42.666695 23721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.672580 23721 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:43055
I20260925 14:57:42.672626 23849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:43055 every 8 connection(s)
I20260925 14:57:42.672991 23721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:42.674957 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23721
I20260925 14:57:42.675051 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:57:42.675242 9781 external_mini_cluster.cc:1428] Setting key cf1e036d9fb07969938137fac85e569b
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:57:42.682987 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.673727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:33157 (local address 127.9.141.126:39829)
0925 14:57:42.673774 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.673776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.674001 (+ 225us) server_negotiation.cc:407] Connection header received
0925 14:57:42.674562 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.674564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.674577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.674628 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.675782 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.676537 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.677794 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.677980 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.678039 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.680294 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.680295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.680296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:42.680303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.681674 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.682072 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.682074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.682074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.682133 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.682376 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.682377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.682377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.682484 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.682550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.682822 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.682899 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:57:42.683172 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.673695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:33157)
0925 14:57:42.673880 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:42.673916 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:42.674448 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:42.674698 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:42.674702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:42.675368 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:42.675679 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.675686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.676623 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.676624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.677690 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.677692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.678064 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.678065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.678229 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.678807 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:42.678815 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:42.680201 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:42.681771 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:42.681778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:42.681780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:42.681998 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:42.682204 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:42.682205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:42.682206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:42.682306 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:42.682552 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:42.682559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:42.682694 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:57:42.682791 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:57:42.682942 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260925 14:57:42.683624 23850 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39829
I20260925 14:57:42.683709 23850 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:42.683842 23850 heartbeater.cc:511] Master 127.9.141.126:39829 requested a full tablet report, sending...
I20260925 14:57:42.684233 23525 ts_manager.cc:194] Registered new tserver with Master: 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
I20260925 14:57:42.684643 23525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:33157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:57:42.699640 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39829
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:42.780082 23855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:42.780292 23855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:42.780315 23855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:42.781605 23855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:42.781651 23855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:42.781677 23855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:57:42.783047 23855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:42.783236 23855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:42.783406 23855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:42.788666 23860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.788654 23863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:42.788653 23861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:42.788908 23855 server_base.cc:1034] running on GCE node
I20260925 14:57:42.789147 23855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:42.789419 23855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:42.790617 23855 hybrid_clock.cc:648] HybridClock initialized: now 1790348262790602 us; error 33 us; skew 500 ppm
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:42.793354 23855 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:57:42.793892 23855 webserver.cc:460] Webserver started at http://127.9.141.67:44061/ using document root <none> and password file <none>
I20260925 14:57:42.794126 23855 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:42.794190 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:42.794334 23855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:57:42.795409 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "df0b47e54bfd4eb29ff03b1079bbe798"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "288efb788f727563513a1c4afa0d424b"
server_key_iv: "4dc0fada723047ad666174d1a13d27d4"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.795782 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "df0b47e54bfd4eb29ff03b1079bbe798"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "288efb788f727563513a1c4afa0d424b"
server_key_iv: "4dc0fada723047ad666174d1a13d27d4"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.797266 23855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:57:42.798058 23870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.798300 23855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:42.798422 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "df0b47e54bfd4eb29ff03b1079bbe798"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "288efb788f727563513a1c4afa0d424b"
server_key_iv: "4dc0fada723047ad666174d1a13d27d4"
server_key_version: "encryptionkey@0"
I20260925 14:57:42.798502 23855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:42.813961 23855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:42.816601 23855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:42.816754 23855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:42.816989 23855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:42.817297 23855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:42.817349 23855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.817391 23855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:42.817417 23855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:42.823383 23855 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:41857
I20260925 14:57:42.823426 23983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:41857 every 8 connection(s)
I20260925 14:57:42.823813 23855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:42.824226 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 23855
I20260925 14:57:42.824324 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:57:42.824556 9781 external_mini_cluster.cc:1428] Setting key 02a4d152a5585f497b103660d0276861
Sep 25 14:57:42 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348262, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:42.833729 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.824561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:41867)
0925 14:57:42.824733 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:42.824784 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:42.825445 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:42.825664 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:42.825668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:42.826244 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:42.826437 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.826442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.827153 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.827154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.828053 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.828056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.828479 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.828480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.828609 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.829082 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:42.829090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:42.830488 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:42.832100 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:42.832106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:42.832108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:42.832332 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:42.832573 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:42.832574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:42.832574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:42.832685 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:42.832944 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:42.832952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:42.833079 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 14:57:42.833185 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:57:42.833478 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260925 14:57:42.833750 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.824581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:41867 (local address 127.9.141.126:39829)
0925 14:57:42.824640 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.824642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.824863 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:57:42.825529 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.825530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.825541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.825585 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.826532 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.827041 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.828155 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.828365 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.828472 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.830582 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.830584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.830584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:42.830591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.832021 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.832450 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.832451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.832451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.832503 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.832751 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.832751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.832752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.832859 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.832913 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.833366 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.833441 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:57:42.834380 23984 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39829
I20260925 14:57:42.834483 23984 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:42.834641 23984 heartbeater.cc:511] Master 127.9.141.126:39829 requested a full tablet report, sending...
I20260925 14:57:42.835018 23525 ts_manager.cc:194] Registered new tserver with Master: df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857)
I20260925 14:57:42.835402 23525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:41867
I20260925 14:57:42.836427 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:42.844362 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.837903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.9.141.126:39829)
0925 14:57:42.837981 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.837984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.838079 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:57:42.838162 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.838163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.838173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.838224 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:42.838803 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.839288 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.840042 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.840243 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.840328 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.841709 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:42.841711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:42.841711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:42.841718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:42.843094 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.843456 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.843457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.843458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.843514 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:42.843713 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:42.843714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:42.843714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:42.843876 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:42.843979 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.844161 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.844254 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:57:42.846331 23525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:42.848217 23525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:42.855759 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.851755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.9.141.65:44033)
0925 14:57:42.851963 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.851968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.851976 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:42.852013 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.852014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.852028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.852103 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:42.852206 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.853362 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.854732 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.855326 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.855415 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.855480 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.855605 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.855648 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260925 14:57:42.855845 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.851203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:41857 (local address 127.0.0.1:39532)
0925 14:57:42.851408 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:42.851451 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:42.851522 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:42.852009 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:42.852011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:42.852015 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:42.852101 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.852104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.853638 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.853639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.854657 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.854660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.855608 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.855609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.855668 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.855671 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:42.855699 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 14:57:42.855732 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260925 14:57:42.855911 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.851377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34484 (local address 127.9.141.66:43055)
0925 14:57:42.851756 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.851761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.851836 (+ 75us) server_negotiation.cc:407] Connection header received
0925 14:57:42.851881 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.851883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.851898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.851976 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:42.852223 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.853021 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.854594 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.855375 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.855453 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.855510 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.855793 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.855833 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":102,"threads_started":1}
I20260925 14:57:42.855983 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.851313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.9.141.67:41857)
0925 14:57:42.851810 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:42.851815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:42.851826 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:57:42.851869 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:42.851871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:42.851886 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:42.852233 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:42.852302 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.853514 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.854769 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.855522 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.855772 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.855834 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:42.855858 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:57:42.855894 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":98,"threads_started":1}
I20260925 14:57:42.855733 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.851450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:44033 (local address 127.0.0.1:42082)
0925 14:57:42.851612 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:42.851621 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:42.851661 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:42.852099 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:42.852100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:42.852101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:42.852148 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.852149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.853469 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.853470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.854639 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.854648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.855415 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.855415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.855503 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.855517 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:57:42.855588 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:57:42.855648 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":23,"thread_start_us":120,"threads_started":1}
I20260925 14:57:42.856141 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:42.851216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:43055 (local address 127.0.0.1:34484)
0925 14:57:42.851562 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:42.851574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:42.851627 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:42.852066 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:42.852066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:42.852068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:42.852138 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.852140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.853122 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.853123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.854483 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:42.854486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:42.855547 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:42.855548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:42.855618 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:42.855622 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:42.855668 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:57:42.855718 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":11,"thread_start_us":81,"threads_started":1}
I20260925 14:57:42.857023 23784 tablet_service.cc:1467] Processing CreateTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 (DEFAULT_TABLE table=test-table [id=1c6fc4111f954a25b957e08eae11cfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:42.857035 23918 tablet_service.cc:1467] Processing CreateTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 (DEFAULT_TABLE table=test-table [id=1c6fc4111f954a25b957e08eae11cfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:42.857178 23650 tablet_service.cc:1467] Processing CreateTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 (DEFAULT_TABLE table=test-table [id=1c6fc4111f954a25b957e08eae11cfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:42.857381 23784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86858cc0185a47a9a8ab0d317bb5ecf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:42.857383 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86858cc0185a47a9a8ab0d317bb5ecf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:42.857456 23650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86858cc0185a47a9a8ab0d317bb5ecf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:42.860006 24001 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Bootstrap starting.
I20260925 14:57:42.860801 24001 tablet_bootstrap.cc:654] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:42.860988 24002 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap starting.
I20260925 14:57:42.861129 24003 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Bootstrap starting.
I20260925 14:57:42.861428 24001 log.cc:824] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:42.861716 24003 tablet_bootstrap.cc:654] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:42.861716 24002 tablet_bootstrap.cc:654] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:42.862195 24001 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: No bootstrap required, opened a new log
I20260925 14:57:42.862267 24001 ts_tablet_manager.cc:1397] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:57:42.862363 24002 log.cc:824] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:42.863109 24002 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: No bootstrap required, opened a new log
I20260925 14:57:42.863106 24003 log.cc:824] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:42.863263 24002 ts_tablet_manager.cc:1397] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:42.864445 24003 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: No bootstrap required, opened a new log
I20260925 14:57:42.864532 24003 ts_tablet_manager.cc:1397] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:57:42.864480 24001 raft_consensus.cc:348] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:42.864669 24001 raft_consensus.cc:374] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:42.864712 24001 raft_consensus.cc:729] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be67d367ec24ab58418a271e15f11cb, State: Initialized, Role: FOLLOWER
I20260925 14:57:42.864843 24001 consensus_queue.cc:260] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:42.865121 24001 ts_tablet_manager.cc:1428] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:57:42.865267 23850 heartbeater.cc:503] Master 127.9.141.126:39829 was elected leader, sending a full tablet report...
I20260925 14:57:42.866137 24002 raft_consensus.cc:348] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:42.866554 24002 raft_consensus.cc:374] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:42.866648 24002 raft_consensus.cc:729] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40498e7071e94bb0ace7c7ee3cc75d7f, State: Initialized, Role: FOLLOWER
I20260925 14:57:42.866822 24002 consensus_queue.cc:260] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:42.867158 24002 ts_tablet_manager.cc:1428] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:57:42.867075 24003 raft_consensus.cc:348] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:42.867169 23716 heartbeater.cc:503] Master 127.9.141.126:39829 was elected leader, sending a full tablet report...
I20260925 14:57:42.867267 24003 raft_consensus.cc:374] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:42.867321 24003 raft_consensus.cc:729] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0b47e54bfd4eb29ff03b1079bbe798, State: Initialized, Role: FOLLOWER
I20260925 14:57:42.867442 24003 consensus_queue.cc:260] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:42.867714 24003 ts_tablet_manager.cc:1428] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:57:42.867880 23984 heartbeater.cc:503] Master 127.9.141.126:39829 was elected leader, sending a full tablet report...
W20260925 14:57:42.923574 23851 tablet.cc:2369] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:43.002143 23717 tablet.cc:2369] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:43.011667 24009 raft_consensus.cc:482] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:43.011787 24009 raft_consensus.cc:504] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:43.012233 24009 leader_election.cc:288] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055), 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033)
I20260925 14:57:43.016162 24010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.012635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:44033 (local address 127.9.141.67:52871)
0925 14:57:43.012831 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.012841 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.012892 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.013051 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.013052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.013053 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:43.013100 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.013101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.014014 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.014014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.015359 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.015362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.015908 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.015910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.015980 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.015984 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.016038 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.016082 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":50,"thread_start_us":106,"threads_started":1}
I20260925 14:57:43.016162 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.012712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:52871 (local address 127.9.141.65:44033)
0925 14:57:43.012753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.012755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.012884 (+ 129us) server_negotiation.cc:407] Connection header received
0925 14:57:43.012976 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.012977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.012987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.013042 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:43.013159 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.013844 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.015441 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.015828 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.015888 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.015895 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.016062 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.016103 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:57:43.016444 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.012528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:49085 (local address 127.9.141.66:43055)
0925 14:57:43.012593 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.012595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.012622 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:57:43.012712 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.012713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.012725 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.012813 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:43.012999 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.013771 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.015462 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.015878 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.015953 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.015961 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.016303 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.016349 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:57:43.016469 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.012493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:43055 (local address 127.9.141.67:49085)
0925 14:57:43.012543 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.012569 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.012643 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.012844 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.012845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.012847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:43.012926 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.012928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.013870 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.013871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.015359 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.015362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.016124 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.016125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.016200 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.016203 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.016294 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.016346 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":93}
I20260925 14:57:43.016654 23670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86858cc0185a47a9a8ab0d317bb5ecf4" candidate_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" is_pre_election: true
I20260925 14:57:43.016819 23670 raft_consensus.cc:2454] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0b47e54bfd4eb29ff03b1079bbe798 in term 0.
I20260925 14:57:43.016944 23804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86858cc0185a47a9a8ab0d317bb5ecf4" candidate_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be67d367ec24ab58418a271e15f11cb" is_pre_election: true
I20260925 14:57:43.017083 23873 leader_election.cc:302] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f, df0b47e54bfd4eb29ff03b1079bbe798; no voters:
I20260925 14:57:43.017117 23804 raft_consensus.cc:2454] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0b47e54bfd4eb29ff03b1079bbe798 in term 0.
I20260925 14:57:43.017215 24009 raft_consensus.cc:2781] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:43.017266 24009 raft_consensus.cc:482] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:43.017289 24009 raft_consensus.cc:3037] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:43.018003 24009 raft_consensus.cc:504] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:43.018134 24009 leader_election.cc:288] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [CANDIDATE]: Term 1 election: Requested vote from peers 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055), 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033)
I20260925 14:57:43.018306 23804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86858cc0185a47a9a8ab0d317bb5ecf4" candidate_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be67d367ec24ab58418a271e15f11cb"
I20260925 14:57:43.018359 23670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86858cc0185a47a9a8ab0d317bb5ecf4" candidate_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f"
I20260925 14:57:43.018399 23804 raft_consensus.cc:3037] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:43.018432 23670 raft_consensus.cc:3037] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:43.019294 23804 raft_consensus.cc:2454] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0b47e54bfd4eb29ff03b1079bbe798 in term 1.
I20260925 14:57:43.019332 23670 raft_consensus.cc:2454] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0b47e54bfd4eb29ff03b1079bbe798 in term 1.
I20260925 14:57:43.019467 23873 leader_election.cc:302] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f, df0b47e54bfd4eb29ff03b1079bbe798; no voters:
I20260925 14:57:43.019567 24009 raft_consensus.cc:2781] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:43.019706 24009 raft_consensus.cc:686] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 LEADER]: Becoming Leader. State: Replica: df0b47e54bfd4eb29ff03b1079bbe798, State: Running, Role: LEADER
I20260925 14:57:43.019800 24009 consensus_queue.cc:237] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:43.020627 23525 catalog_manager.cc:5495] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 reported cstate change: term changed from 0 to 1, leader changed from <none> to df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } health_report { overall_health: HEALTHY } } }
I20260925 14:57:43.037910 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.035346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.9.141.67:41857)
0925 14:57:43.035416 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.035418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.035428 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:43.035535 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.035536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.035553 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.035627 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:43.035764 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.036221 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.037099 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.037291 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.037372 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.037555 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:43.037651 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:43.037734 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.037781 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.037800 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:57:43.039582 23804 raft_consensus.cc:1261] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Refusing update from remote peer df0b47e54bfd4eb29ff03b1079bbe798: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:43.039580 23670 raft_consensus.cc:1261] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Refusing update from remote peer df0b47e54bfd4eb29ff03b1079bbe798: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:43.039930 24011 consensus_queue.cc:1035] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:43.040060 24009 consensus_queue.cc:1035] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:43.043375 24015 mvcc.cc:205] Tried to move back new op lower bound from 7333266485407543296 to 7333266485329719296. Current Snapshot: MvccSnapshot[applied={T|T < 7333266485407543296}]
I20260925 14:57:43.043834 24016 mvcc.cc:205] Tried to move back new op lower bound from 7333266485407543296 to 7333266485329719296. Current Snapshot: MvccSnapshot[applied={T|T < 7333266485407543296 or (T in {7333266485407543296})}]
I20260925 14:57:43.044716 23525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:43.044812 23525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:43.045564 23525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Sending DeleteTablet for 3 replicas of tablet 86858cc0185a47a9a8ab0d317bb5ecf4
I20260925 14:57:43.045835 23918 tablet_service.cc:1514] Processing DeleteTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:43 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:39532
I20260925 14:57:43.045933 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23587
I20260925 14:57:43.045943 23649 tablet_service.cc:1514] Processing DeleteTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:43 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:42082
I20260925 14:57:43.046420 23784 tablet_service.cc:1514] Processing DeleteTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:43 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:34484
I20260925 14:57:43.046622 24025 tablet_replica.cc:330] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: stopping tablet replica
I20260925 14:57:43.046766 24025 raft_consensus.cc:2229] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:43.046900 24025 raft_consensus.cc:2258] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:43.047262 24024 tablet_replica.cc:330] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: stopping tablet replica
I20260925 14:57:43.047374 24024 raft_consensus.cc:2229] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:57:43.047398 24025 ts_tablet_manager.cc:1905] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:43.048559 24024 raft_consensus.cc:2258] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:43.049027 24024 ts_tablet_manager.cc:1905] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:43.049635 24025 ts_tablet_manager.cc:1918] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:43.049770 24025 log.cc:1196] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/86858cc0185a47a9a8ab0d317bb5ecf4
I20260925 14:57:43.050135 24025 ts_tablet_manager.cc:1939] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 0be67d367ec24ab58418a271e15f11cb: Deleting consensus metadata
I20260925 14:57:43.050613 23511 catalog_manager.cc:4841] TS 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055): tablet 86858cc0185a47a9a8ab0d317bb5ecf4 (table test-table [id=1c6fc4111f954a25b957e08eae11cfea]) successfully deleted
I20260925 14:57:43.050688 24024 ts_tablet_manager.cc:1918] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:43.050750 24024 log.cc:1196] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/86858cc0185a47a9a8ab0d317bb5ecf4
I20260925 14:57:43.051038 24024 ts_tablet_manager.cc:1939] T 86858cc0185a47a9a8ab0d317bb5ecf4 P df0b47e54bfd4eb29ff03b1079bbe798: Deleting consensus metadata
I20260925 14:57:43.051501 23510 catalog_manager.cc:4841] TS df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): tablet 86858cc0185a47a9a8ab0d317bb5ecf4 (table test-table [id=1c6fc4111f954a25b957e08eae11cfea]) successfully deleted
W20260925 14:57:43.052270 23511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:44033 (error 108)
W20260925 14:57:43.052915 23998 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:43.052760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:44033 (local address 127.0.0.1:42098)
0925 14:57:43.052816 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.052828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.052866 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:44033: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:57:43.053013 23511 catalog_manager.cc:4584] TS 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86858cc0185a47a9a8ab0d317bb5ecf4: Network error: Client connection negotiation failed: client connection to 127.9.141.65:44033: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 14:57:43.053341 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23721
I20260925 14:57:43.060022 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23855
I20260925 14:57:43.066037 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 23493
I20260925 14:57:43.071236 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39829
--webserver_interface=127.9.141.126
--webserver_port=34031
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:39829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:43.153391 24027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:43.153609 24027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:43.153631 24027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:43.154820 24027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:43.154865 24027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:43.154879 24027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:43.154892 24027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:43.154904 24027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:43.156487 24027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:39829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=34031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:43.156746 24027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:43.156986 24027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:43.165642 24033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.165642 24036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.165665 24034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.165956 24027 server_base.cc:1034] running on GCE node
I20260925 14:57:43.166340 24027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:43.166637 24027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:43.167786 24027 hybrid_clock.cc:648] HybridClock initialized: now 1790348263167762 us; error 34 us; skew 500 ppm
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:43.175601 24027 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:43.176084 24027 webserver.cc:460] Webserver started at http://127.9.141.126:34031/ using document root <none> and password file <none>
I20260925 14:57:43.176349 24027 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:43.176463 24027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:43.177886 24027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:43.178610 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.178821 24027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:43.178936 24027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "43b5c758a22e419b8e42d4afd3961f24"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "7cc39e5b0aebd28ef5c25bfb249ce129"
server_key_iv: "b26f5a7edb3d2e93482fb4f2a47de3aa"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.179391 24027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:43.196570 24027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:43.198539 24027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:43.198691 24027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:43.203106 24027 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:39829
I20260925 14:57:43.203138 24095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:39829 every 8 connection(s)
I20260925 14:57:43.203552 24027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:43.205164 24096 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:57:43.205760 24096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap starting.
I20260925 14:57:43.206003 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24027
I20260925 14:57:43.206250 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:44033
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:39829
--webserver_port=46495
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:43.214910 24096 log.cc:824] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:43.216873 24096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:43.217170 24096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap complete.
I20260925 14:57:43.219385 24096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:43.219612 24096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b5c758a22e419b8e42d4afd3961f24, State: Initialized, Role: FOLLOWER
I20260925 14:57:43.219786 24096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:43.219841 24096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:43.219905 24096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:43.219949 24096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:57:43.221009 24096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:43.221095 24096 leader_election.cc:302] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b5c758a22e419b8e42d4afd3961f24; no voters:
I20260925 14:57:43.221310 24096 leader_election.cc:288] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:57:43.221345 24100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:57:43.221508 24100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 LEADER]: Becoming Leader. State: Replica: 43b5c758a22e419b8e42d4afd3961f24, State: Running, Role: LEADER
I20260925 14:57:43.221580 24096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:43.221657 24100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:43.221922 24100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b5c758a22e419b8e42d4afd3961f24. Latest consensus state: current_term: 2 leader_uuid: "43b5c758a22e419b8e42d4afd3961f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } } }
I20260925 14:57:43.222008 24100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:43.221928 24101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43b5c758a22e419b8e42d4afd3961f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } } }
I20260925 14:57:43.222256 24101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:43.222757 24106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:43.223556 24106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:43.223759 24106 catalog_manager.cc:1259] Loaded cluster ID: e40de25767e3433f89fe0cbbde430631
I20260925 14:57:43.223958 24106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:57:43.224243 24106 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 14:57:43.228940 24106 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 14:57:43.231359 24106 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:57:43.231561 24106 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Loaded TSK: 0
I20260925 14:57:43.231791 24106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:57:43.290632 24098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:43.290868 24098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:43.290910 24098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:43.292151 24098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:43.292222 24098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:43.292269 24098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:57:43.293840 24098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=46495
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:43.294147 24098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:43.294409 24098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:43.299726 24123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.299739 24124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.299814 24126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.300074 24098 server_base.cc:1034] running on GCE node
I20260925 14:57:43.300297 24098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:43.300616 24098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:43.301800 24098 hybrid_clock.cc:648] HybridClock initialized: now 1790348263301777 us; error 35 us; skew 500 ppm
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:43.304749 24098 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:57:43.305289 24098 webserver.cc:460] Webserver started at http://127.9.141.65:46495/ using document root <none> and password file <none>
I20260925 14:57:43.305562 24098 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:43.305660 24098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:43.307516 24098 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:57:43.308301 24133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.308553 24098 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:43.308656 24098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "40498e7071e94bb0ace7c7ee3cc75d7f"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "57870a5dd6b7a0ded1b6e129bd8a5ae6"
server_key_iv: "684a3dee8d102f843963f0d8b91e070c"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.309094 24098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:43.330492 24098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:43.333329 24098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:43.333495 24098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:43.333777 24098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:43.334331 24140 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:57:43.335968 24098 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:57:43.336027 24098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:57:43.336063 24098 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:57:43.336971 24098 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:57:43.337016 24098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:43.337105 24140 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap starting.
I20260925 14:57:43.344002 24098 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:44033
I20260925 14:57:43.344226 24247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:44033 every 8 connection(s)
I20260925 14:57:43.344476 24098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:43.348110 24140 log.cc:824] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:43.350638 24140 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:43.350924 24140 tablet_bootstrap.cc:492] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap complete.
I20260925 14:57:43.351933 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24098
I20260925 14:57:43.352150 24140 ts_tablet_manager.cc:1397] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20260925 14:57:43.352192 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:43055
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:39829
--webserver_port=34997
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:43.353966 24140 raft_consensus.cc:348] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:43.354327 24140 raft_consensus.cc:729] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40498e7071e94bb0ace7c7ee3cc75d7f, State: Initialized, Role: FOLLOWER
I20260925 14:57:43.354458 24140 consensus_queue.cc:260] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } } peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } }
I20260925 14:57:43.354763 24140 ts_tablet_manager.cc:1428] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent starting tablet: real 0.003s user 0.004s sys 0.002s
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:43.359998 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.345195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:34351)
0925 14:57:43.345413 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.345468 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.346180 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.346703 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.346708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.347099 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:43.347433 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.347442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.349119 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.349120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.350344 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.350346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.351180 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.351181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.351344 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.353880 (+ 2536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:43.353891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:43.355790 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:43.358239 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.358247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.358250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.358520 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.358824 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.358825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.358825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.358937 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.359338 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:43.359345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:43.359485 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.359597 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.359790 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260925 14:57:43.360023 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.345559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:34351 (local address 127.9.141.126:39829)
0925 14:57:43.345756 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.345766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.345808 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:57:43.346407 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.346414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.346470 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.346624 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.347584 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.349001 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.350441 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.351058 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.351173 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.356066 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.356070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.356070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:43.356083 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.358098 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.358628 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.358631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.358632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.358740 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.359005 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.359006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.359007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.359262 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.359361 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.359653 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.359790 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260925 14:57:43.360975 24248 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39829
I20260925 14:57:43.361088 24248 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:43.361284 24248 heartbeater.cc:511] Master 127.9.141.126:39829 requested a full tablet report, sending...
I20260925 14:57:43.362042 24059 ts_manager.cc:194] Registered new tserver with Master: 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033)
I20260925 14:57:43.363145 24059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:34351
I20260925 14:57:43.363965 24248 heartbeater.cc:503] Master 127.9.141.126:39829 was elected leader, sending a full tablet report...
I20260925 14:57:43.371719 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.362854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.9.141.65:44033)
0925 14:57:43.368804 (+ 5950us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.368809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.368819 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:43.368855 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.368857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.368872 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.368969 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:43.369142 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.369810 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.370824 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.371300 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.371396 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.371453 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.371609 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.371647 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5908,"thread_start_us":62,"threads_started":1}
I20260925 14:57:43.371719 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.362608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:44033 (local address 127.0.0.1:42102)
0925 14:57:43.368527 (+ 5919us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.368568 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.368628 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.368983 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.368987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.368989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:43.369054 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.369056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.369892 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.369893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.370727 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.370737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.371399 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.371400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.371484 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.371496 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.371591 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.371647 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5889,"thread_start_us":94,"threads_started":1}
I20260925 14:57:43.372434 24182 tablet_service.cc:1514] Processing DeleteTablet for tablet 86858cc0185a47a9a8ab0d317bb5ecf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:43 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:42102
I20260925 14:57:43.372946 24257 tablet_replica.cc:330] stopping tablet replica
I20260925 14:57:43.373062 24257 raft_consensus.cc:2229] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:43.373131 24257 raft_consensus.cc:2258] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:43.373435 24257 ts_tablet_manager.cc:1905] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:43.375439 24257 ts_tablet_manager.cc:1918] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:43.375619 24257 log.cc:1196] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/86858cc0185a47a9a8ab0d317bb5ecf4
I20260925 14:57:43.376083 24257 ts_tablet_manager.cc:1939] T 86858cc0185a47a9a8ab0d317bb5ecf4 P 40498e7071e94bb0ace7c7ee3cc75d7f: Deleting consensus metadata
I20260925 14:57:43.377331 24046 catalog_manager.cc:4841] TS 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033): tablet 86858cc0185a47a9a8ab0d317bb5ecf4 (table test-table [id=1c6fc4111f954a25b957e08eae11cfea]) successfully deleted
W20260925 14:57:43.442569 24253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:43.442808 24253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:43.442833 24253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:43.444072 24253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:43.444123 24253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:43.444151 24253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:57:43.445703 24253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:43055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=34997
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:43.445947 24253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:43.446254 24253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:43.451535 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.451543 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.451535 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.451850 24253 server_base.cc:1034] running on GCE node
I20260925 14:57:43.452070 24253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:43.452337 24253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:43.453517 24253 hybrid_clock.cc:648] HybridClock initialized: now 1790348263453495 us; error 39 us; skew 500 ppm
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:43.456130 24253 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:57:43.456614 24253 webserver.cc:460] Webserver started at http://127.9.141.66:34997/ using document root <none> and password file <none>
I20260925 14:57:43.456840 24253 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:43.456952 24253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:43.458346 24253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:43.458940 24272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.459103 24253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.459203 24253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "0be67d367ec24ab58418a271e15f11cb"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "e5342947b59a5343b9ab1dd0e2747cb1"
server_key_iv: "5140847d0b94f77afa7efb10dcb00608"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.459553 24253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:43.466941 24253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:43.469234 24253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:43.469393 24253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:43.469624 24253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:43.469925 24253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:43.469975 24253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.470018 24253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:43.470043 24253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.476491 24253 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:43055
I20260925 14:57:43.476553 24385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:43055 every 8 connection(s)
I20260925 14:57:43.476902 24253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:43.479002 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24253
I20260925 14:57:43.479279 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:41857
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:39829
--webserver_port=44061
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:43.488957 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.478585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:57549 (local address 127.9.141.126:39829)
0925 14:57:43.478633 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.478634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.478872 (+ 238us) server_negotiation.cc:407] Connection header received
0925 14:57:43.479585 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.479586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.479603 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.479687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.480760 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.481265 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.482673 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.483055 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.483129 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.485478 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.485479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.485480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:43.485489 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.487334 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.487767 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.487769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.487769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.487838 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.488136 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.488137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.488137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.488304 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.488400 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.488709 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.488818 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:57:43.488979 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.478574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:57549)
0925 14:57:43.478745 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.478785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.479482 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.479751 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.479754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.480324 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:43.480631 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.480639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.481397 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.481398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.482573 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.482576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.483163 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.483164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.483327 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.483899 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:43.483908 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:43.485380 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:43.487433 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.487440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.487442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.487686 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.487945 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.487946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.487946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.488061 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.488426 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:43.488433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:43.488569 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.488662 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.488806 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260925 14:57:43.489547 24386 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39829
I20260925 14:57:43.489662 24386 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:43.489846 24386 heartbeater.cc:511] Master 127.9.141.126:39829 requested a full tablet report, sending...
I20260925 14:57:43.490259 24059 ts_manager.cc:194] Registered new tserver with Master: 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
I20260925 14:57:43.490792 24059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:57549
W20260925 14:57:43.568182 24389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:43.568481 24389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:43.568513 24389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:43.569864 24389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:43.569914 24389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:43.569941 24389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:57:43.571385 24389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:41857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=44061
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:43.571578 24389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:43.571754 24389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:43.576889 24394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.576884 24395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:43.576879 24397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:43.577219 24389 server_base.cc:1034] running on GCE node
I20260925 14:57:43.577462 24389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:43.577736 24389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:43.578905 24389 hybrid_clock.cc:648] HybridClock initialized: now 1790348263578884 us; error 36 us; skew 500 ppm
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:43.581588 24389 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:57:43.582085 24389 webserver.cc:460] Webserver started at http://127.9.141.67:44061/ using document root <none> and password file <none>
I20260925 14:57:43.582290 24389 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:43.582345 24389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:43.583722 24389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:43.584328 24404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.584523 24389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:43.584635 24389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "df0b47e54bfd4eb29ff03b1079bbe798"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "288efb788f727563513a1c4afa0d424b"
server_key_iv: "4dc0fada723047ad666174d1a13d27d4"
server_key_version: "encryptionkey@0"
I20260925 14:57:43.584980 24389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:43.606150 24389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:43.608072 24389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:43.608238 24389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:43.608521 24389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:43.608825 24389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:43.608873 24389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.608914 24389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:43.608942 24389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.614845 24389 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:41857
I20260925 14:57:43.614884 24517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:41857 every 8 connection(s)
I20260925 14:57:43.615307 24389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:57:43 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348263, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:57:43.625310 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24389
I20260925 14:57:43.625386 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.616129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:40421 (local address 127.9.141.126:39829)
0925 14:57:43.616212 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.616215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.616427 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:57:43.616971 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.616972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.616983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.617036 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.618028 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.618557 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.619564 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.619765 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.619851 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.622165 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.622167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.622167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:43.622174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.623821 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.624234 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.624235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.624235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.624291 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.624600 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.624601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.624601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.624729 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.624795 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.625092 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.625277 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:57:43.625454 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.616043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:40421)
0925 14:57:43.616260 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.616301 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.616864 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.617122 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.617127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.617728 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:43.617922 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.617927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.618657 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.618659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.619467 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.619470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.619863 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.619864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.619989 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.620522 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:43.620531 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:43.622061 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:43.623904 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.623911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.623913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.624143 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.624355 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:43.624355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:43.624356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:43.624513 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:43.624813 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:43.624819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:43.624953 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.625041 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.625244 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
I20260925 14:57:43.626272 24518 heartbeater.cc:347] Connected to a master server at 127.9.141.126:39829
I20260925 14:57:43.626389 24518 heartbeater.cc:464] Registering TS with master...
I20260925 14:57:43.626586 24518 heartbeater.cc:511] Master 127.9.141.126:39829 requested a full tablet report, sending...
I20260925 14:57:43.627022 24059 ts_manager.cc:194] Registered new tserver with Master: df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857)
I20260925 14:57:43.627403 24059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:40421
I20260925 14:57:43.632169 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.625903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.9.141.126:39829)
0925 14:57:43.625959 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.625961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.625968 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:57:43.626002 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.626003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.626013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.626075 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:43.626927 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.627484 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.628271 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.628518 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.628576 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.629753 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:43.629755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:43.629755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:43.629761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:43.631209 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.631581 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.631581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.631582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.631638 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:43.631804 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:43.631804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:43.631805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:43.631899 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:43.632009 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.632042 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.632108 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:57:43.632889 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:57:43.635763 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.633252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.9.141.126:39829)
0925 14:57:43.633312 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.633314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.633448 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:57:43.633532 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.633533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.633569 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.633642 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:43.633757 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.634241 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.635014 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.635215 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.635286 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.635451 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:43.635552 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:43.635644 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.635662 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.635686 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:57:43.636042 24059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:57:43.637826 24059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:57:43.640291 24182 tablet_service.cc:1467] Processing CreateTablet for tablet 1656de6753b749588ad2cffcf6129949 (DEFAULT_TABLE table=test-table [id=c4a677a627804547924e925fb1ab2e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:43.640547 24182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1656de6753b749588ad2cffcf6129949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:43.642031 24140 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap starting.
I20260925 14:57:43.642400 24140 tablet_bootstrap.cc:654] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:43.642967 24140 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: No bootstrap required, opened a new log
I20260925 14:57:43.643034 24140 ts_tablet_manager.cc:1397] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:43.643148 24140 raft_consensus.cc:348] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:43.643226 24140 raft_consensus.cc:374] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:43.643275 24140 raft_consensus.cc:729] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40498e7071e94bb0ace7c7ee3cc75d7f, State: Initialized, Role: FOLLOWER
I20260925 14:57:43.643328 24140 consensus_queue.cc:260] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:43.643419 24140 ts_tablet_manager.cc:1428] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:43.644129 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.639781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:43055 (local address 127.0.0.1:34490)
0925 14:57:43.639911 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.639922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.639974 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.640596 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.640597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.640599 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:43.640640 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.640641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.641850 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.641851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.642792 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.642795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.643824 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.643825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.643902 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.643906 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.643982 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.644038 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260925 14:57:43.644155 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.640146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.9.141.66:43055)
0925 14:57:43.640347 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.640351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.640360 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:57:43.640437 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.640439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.640454 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.640530 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:43.640723 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.641749 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.642900 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.643670 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.643721 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.643764 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.644014 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.644067 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260925 14:57:43.644129 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.639698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:41857 (local address 127.0.0.1:39554)
0925 14:57:43.639769 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:43.639789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:43.639832 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:43.640159 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:43.640160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:43.640162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:43.640200 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.640201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.641740 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.641741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.642878 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:43.642881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.643900 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.643900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:43.643959 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.643962 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:43.644010 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:57:43.644050 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":5}
I20260925 14:57:43.644564 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:43.639780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.9.141.67:41857)
0925 14:57:43.639981 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:43.639986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:43.639996 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:57:43.640039 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:43.640042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:43.640068 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:43.640150 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:43.640265 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.641624 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.643010 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:43.643818 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:43.643891 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:43.643951 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:43.644173 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:57:43.644226 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260925 14:57:43.644879 24320 tablet_service.cc:1467] Processing CreateTablet for tablet 1656de6753b749588ad2cffcf6129949 (DEFAULT_TABLE table=test-table [id=c4a677a627804547924e925fb1ab2e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:43.645149 24320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1656de6753b749588ad2cffcf6129949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:43.645129 24452 tablet_service.cc:1467] Processing CreateTablet for tablet 1656de6753b749588ad2cffcf6129949 (DEFAULT_TABLE table=test-table [id=c4a677a627804547924e925fb1ab2e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:57:43.645398 24452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1656de6753b749588ad2cffcf6129949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:57:43.648180 24528 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Bootstrap starting.
I20260925 14:57:43.648082 24527 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Bootstrap starting.
I20260925 14:57:43.649041 24527 tablet_bootstrap.cc:654] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:43.649045 24528 tablet_bootstrap.cc:654] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Neither blocks nor log segments found. Creating new log.
I20260925 14:57:43.649533 24527 log.cc:824] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:43.649533 24528 log.cc:824] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:43.650337 24527 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: No bootstrap required, opened a new log
I20260925 14:57:43.650386 24528 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: No bootstrap required, opened a new log
I20260925 14:57:43.650393 24527 ts_tablet_manager.cc:1397] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:57:43.650453 24528 ts_tablet_manager.cc:1397] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:43.651799 24527 raft_consensus.cc:348] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:43.651818 24528 raft_consensus.cc:348] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:43.651990 24527 raft_consensus.cc:374] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:43.651990 24528 raft_consensus.cc:374] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:57:43.652032 24528 raft_consensus.cc:729] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be67d367ec24ab58418a271e15f11cb, State: Initialized, Role: FOLLOWER
I20260925 14:57:43.652034 24527 raft_consensus.cc:729] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0b47e54bfd4eb29ff03b1079bbe798, State: Initialized, Role: FOLLOWER
I20260925 14:57:43.652153 24528 consensus_queue.cc:260] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:43.652174 24527 consensus_queue.cc:260] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:43.652498 24528 ts_tablet_manager.cc:1428] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:57:43.652504 24527 ts_tablet_manager.cc:1428] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:57:43.652510 24518 heartbeater.cc:503] Master 127.9.141.126:39829 was elected leader, sending a full tablet report...
I20260925 14:57:43.652655 24386 heartbeater.cc:503] Master 127.9.141.126:39829 was elected leader, sending a full tablet report...
W20260925 14:57:43.727537 24387 tablet.cc:2369] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:43.848893 24249 tablet.cc:2369] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:57:43.865973 24519 tablet.cc:2369] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:57:44.003681 24254 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:44.003834 24254 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:44.004199 24254 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
I20260925 14:57:44.007743 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.004505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:34947 (local address 127.9.141.66:43055)
0925 14:57:44.004597 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.004599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.004693 (+ 94us) server_negotiation.cc:407] Connection header received
0925 14:57:44.004778 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.004779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.004826 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.004886 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:44.005043 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.005530 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.007017 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.007385 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.007473 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.007482 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.007630 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.007670 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 14:57:44.007751 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.004465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:43055 (local address 127.9.141.65:34947)
0925 14:57:44.004603 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.004613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.004707 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.004907 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.004908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.004911 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:44.004976 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.004978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.005621 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.005622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.006886 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.006888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.007473 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.007474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.007543 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.007546 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.007620 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.007667 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":38,"thread_start_us":62,"threads_started":1}
I20260925 14:57:44.007880 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.004454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:41857 (local address 127.9.141.65:44827)
0925 14:57:44.004579 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.004591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.004664 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.004873 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:44.004875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:44.004877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:57:44.004940 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.004942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.005574 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.005575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.006886 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:44.006888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.007562 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.007563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:44.007640 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.007643 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:57:44.007705 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:57:44.007752 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":34,"thread_start_us":72,"threads_started":1}
I20260925 14:57:44.007867 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.004528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:44827 (local address 127.9.141.67:41857)
0925 14:57:44.004588 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.004591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.004646 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:57:44.004740 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.004741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.004755 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.004864 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:57:44.005009 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.005480 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.007016 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.007442 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.007549 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.007557 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.007724 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.007771 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:57:44.008270 24340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1656de6753b749588ad2cffcf6129949" candidate_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be67d367ec24ab58418a271e15f11cb" is_pre_election: true
I20260925 14:57:44.008270 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1656de6753b749588ad2cffcf6129949" candidate_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" is_pre_election: true
I20260925 14:57:44.008466 24472 raft_consensus.cc:2454] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40498e7071e94bb0ace7c7ee3cc75d7f in term 0.
I20260925 14:57:44.008466 24340 raft_consensus.cc:2454] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40498e7071e94bb0ace7c7ee3cc75d7f in term 0.
I20260925 14:57:44.008708 24136 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0be67d367ec24ab58418a271e15f11cb, 40498e7071e94bb0ace7c7ee3cc75d7f, df0b47e54bfd4eb29ff03b1079bbe798; no voters:
I20260925 14:57:44.008880 24254 raft_consensus.cc:2781] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:57:44.008987 24254 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:57:44.009021 24254 raft_consensus.cc:3037] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:44.009766 24254 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:44.009930 24254 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 1 election: Requested vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
I20260925 14:57:44.010077 24472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1656de6753b749588ad2cffcf6129949" candidate_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0b47e54bfd4eb29ff03b1079bbe798"
I20260925 14:57:44.010111 24340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1656de6753b749588ad2cffcf6129949" candidate_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0be67d367ec24ab58418a271e15f11cb"
I20260925 14:57:44.010159 24472 raft_consensus.cc:3037] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:44.010190 24340 raft_consensus.cc:3037] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:57:44.011035 24472 raft_consensus.cc:2454] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40498e7071e94bb0ace7c7ee3cc75d7f in term 1.
I20260925 14:57:44.011045 24340 raft_consensus.cc:2454] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40498e7071e94bb0ace7c7ee3cc75d7f in term 1.
I20260925 14:57:44.011245 24135 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f, df0b47e54bfd4eb29ff03b1079bbe798; no voters:
I20260925 14:57:44.011379 24254 raft_consensus.cc:2781] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:57:44.011507 24254 raft_consensus.cc:686] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 LEADER]: Becoming Leader. State: Replica: 40498e7071e94bb0ace7c7ee3cc75d7f, State: Running, Role: LEADER
I20260925 14:57:44.011580 24254 consensus_queue.cc:237] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:44.012348 24060 catalog_manager.cc:5495] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65). New cstate: current_term: 1 leader_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } health_report { overall_health: HEALTHY } } }
I20260925 14:57:44.087193 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.084484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.9.141.65:44033)
0925 14:57:44.084638 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:44.084640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:44.084648 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:57:44.084693 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:44.084694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:44.084716 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:44.084802 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:57:44.084892 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.085410 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.086146 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:44.086383 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:44.086441 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:44.086623 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:57:44.086734 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:57:44.086795 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:44.086840 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:57:44.086856 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260925 14:57:44.088901 24340 raft_consensus.cc:1261] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Refusing update from remote peer 40498e7071e94bb0ace7c7ee3cc75d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:44.088960 24472 raft_consensus.cc:1261] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Refusing update from remote peer 40498e7071e94bb0ace7c7ee3cc75d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:57:44.089258 24535 consensus_queue.cc:1035] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:44.089362 24254 consensus_queue.cc:1035] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:57:44.091787 24541 mvcc.cc:205] Tried to move back new op lower bound from 7333266489705201664 to 7333266489391972352. Current Snapshot: MvccSnapshot[applied={T|T < 7333266489705201664}]
I20260925 14:57:44.092746 24540 mvcc.cc:205] Tried to move back new op lower bound from 7333266489705201664 to 7333266489391972352. Current Snapshot: MvccSnapshot[applied={T|T < 7333266489705201664}]
I20260925 14:57:44.094084 24060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:44.094164 24060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:57:44.094811 24060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Sending DeleteTablet for 3 replicas of tablet 1656de6753b749588ad2cffcf6129949
I20260925 14:57:44.095160 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24098
I20260925 14:57:44.095149 24452 tablet_service.cc:1514] Processing DeleteTablet for tablet 1656de6753b749588ad2cffcf6129949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:44 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:39554
I20260925 14:57:44.095412 24320 tablet_service.cc:1514] Processing DeleteTablet for tablet 1656de6753b749588ad2cffcf6129949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:57:44 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:34490
I20260925 14:57:44.095925 24548 tablet_replica.cc:330] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: stopping tablet replica
I20260925 14:57:44.096055 24548 raft_consensus.cc:2229] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:44.096163 24548 raft_consensus.cc:2258] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:44.096575 24548 ts_tablet_manager.cc:1905] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:44.098588 24548 ts_tablet_manager.cc:1918] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:44.098667 24548 log.cc:1196] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/1656de6753b749588ad2cffcf6129949
I20260925 14:57:44.098913 24548 ts_tablet_manager.cc:1939] T 1656de6753b749588ad2cffcf6129949 P df0b47e54bfd4eb29ff03b1079bbe798: Deleting consensus metadata
I20260925 14:57:44.099356 24045 catalog_manager.cc:4841] TS df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): tablet 1656de6753b749588ad2cffcf6129949 (table test-table [id=c4a677a627804547924e925fb1ab2e4d]) successfully deleted
I20260925 14:57:44.100497 24549 tablet_replica.cc:330] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: stopping tablet replica
I20260925 14:57:44.100616 24549 raft_consensus.cc:2229] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:57:44.100730 24549 raft_consensus.cc:2258] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:57:44.101097 24549 ts_tablet_manager.cc:1905] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:57:44.102178 24046 connection.cc:535] client connection to 127.9.141.65:44033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:57:44.102265 24046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:57:44.102736 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.102591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:44033 (local address 127.0.0.1:42126)
0925 14:57:44.102647 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.102694 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:44033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:57:44.102730 24549 ts_tablet_manager.cc:1918] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:44.102813 24549 log.cc:1196] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/1656de6753b749588ad2cffcf6129949
I20260925 14:57:44.102926 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24253
W20260925 14:57:44.102885 24046 catalog_manager.cc:4584] TS 40498e7071e94bb0ace7c7ee3cc75d7f (127.9.141.65:44033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1656de6753b749588ad2cffcf6129949: Network error: Client connection negotiation failed: client connection to 127.9.141.65:44033: connect: Connection refused (error 111)
I20260925 14:57:44.108079 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.107957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:43055 (local address 127.0.0.1:34506)
0925 14:57:44.108010 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.108039 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.66:43055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:57:44.108595 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24389
I20260925 14:57:44.114121 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24027
I20260925 14:57:44.119302 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39829
--webserver_interface=127.9.141.126
--webserver_port=34031
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:39829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:44.201174 24551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.201395 24551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.201417 24551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.202593 24551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:57:44.202638 24551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.202653 24551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.202666 24551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:57:44.202713 24551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:57:44.204241 24551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:39829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=34031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:44.204535 24551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.204793 24551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:44.209928 24557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.209987 24556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.210124 24559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.210297 24551 server_base.cc:1034] running on GCE node
I20260925 14:57:44.210568 24551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.210877 24551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.212031 24551 hybrid_clock.cc:648] HybridClock initialized: now 1790348264212002 us; error 42 us; skew 500 ppm
Sep 25 14:57:44 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.214679 24551 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:57:44.215164 24551 webserver.cc:460] Webserver started at http://127.9.141.126:34031/ using document root <none> and password file <none>
I20260925 14:57:44.215431 24551 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.215520 24551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.216974 24551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:57:44.217662 24566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.217845 24551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:57:44.217947 24551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "43b5c758a22e419b8e42d4afd3961f24"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "7cc39e5b0aebd28ef5c25bfb249ce129"
server_key_iv: "b26f5a7edb3d2e93482fb4f2a47de3aa"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.218331 24551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.229067 24551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.231141 24551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.231340 24551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.235504 24551 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:39829
I20260925 14:57:44.235549 24618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:39829 every 8 connection(s)
I20260925 14:57:44.235931 24551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:57:44.237663 24619 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:57:44.238389 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap starting.
I20260925 14:57:44.244345 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24551
I20260925 14:57:44.244707 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:44033
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:39829
--webserver_port=46495
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:44.246658 24619 log.cc:824] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:44.249048 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:44.249349 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24: Bootstrap complete.
I20260925 14:57:44.251617 24619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:44.251868 24619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b5c758a22e419b8e42d4afd3961f24, State: Initialized, Role: FOLLOWER
I20260925 14:57:44.252038 24619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:44.252112 24619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:57:44.252164 24619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:57:44.252213 24619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:57:44.253269 24619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:44.253388 24619 leader_election.cc:302] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b5c758a22e419b8e42d4afd3961f24; no voters:
I20260925 14:57:44.253571 24619 leader_election.cc:288] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:57:44.253607 24623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:57:44.253708 24623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [term 3 LEADER]: Becoming Leader. State: Replica: 43b5c758a22e419b8e42d4afd3961f24, State: Running, Role: LEADER
I20260925 14:57:44.253803 24623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } }
I20260925 14:57:44.253850 24619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:57:44.254065 24623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b5c758a22e419b8e42d4afd3961f24. Latest consensus state: current_term: 3 leader_uuid: "43b5c758a22e419b8e42d4afd3961f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } } }
I20260925 14:57:44.254132 24623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:44.254098 24625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "43b5c758a22e419b8e42d4afd3961f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b5c758a22e419b8e42d4afd3961f24" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39829 } } }
I20260925 14:57:44.254297 24625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b5c758a22e419b8e42d4afd3961f24 [sys.catalog]: This master's current role is: LEADER
I20260925 14:57:44.255069 24627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:57:44.256085 24627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:57:44.256189 24627 catalog_manager.cc:1259] Loaded cluster ID: e40de25767e3433f89fe0cbbde430631
I20260925 14:57:44.256222 24627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 14:57:44.256448 24627 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f611cd5b9ad google::LogMessage::Fail() at ??:0
@ 0x7f611cd5e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f611cd5b4de google::LogMessage::Flush() at ??:0
@ 0x7f611cd5ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6124c7c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6124c858b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f611ec290d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f611ec1e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f611fa826db start_thread at ??:0
@ 0x7f611b85961f clone at ??:0
W20260925 14:57:44.368278 24621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.368584 24621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.368624 24621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.369972 24621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.370041 24621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.370090 24621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:57:44.371630 24621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=46495
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:44.371886 24621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.372139 24621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:44.377197 24655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.377223 24656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.377616 24621 server_base.cc:1034] running on GCE node
W20260925 14:57:44.377195 24658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.377861 24621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.378132 24621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.379280 24621 hybrid_clock.cc:648] HybridClock initialized: now 1790348264379258 us; error 35 us; skew 500 ppm
Sep 25 14:57:44 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.382969 24621 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:57:44.383424 24621 webserver.cc:460] Webserver started at http://127.9.141.65:46495/ using document root <none> and password file <none>
I20260925 14:57:44.383654 24621 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.383726 24621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.385110 24621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:44.385798 24665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.386005 24621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:44.386096 24621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "40498e7071e94bb0ace7c7ee3cc75d7f"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "57870a5dd6b7a0ded1b6e129bd8a5ae6"
server_key_iv: "684a3dee8d102f843963f0d8b91e070c"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.386440 24621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.414702 24621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.416950 24621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.417110 24621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.417348 24621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:44.417807 24672 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:57:44.419895 24621 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:57:44.419984 24621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:57:44.420044 24621 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:57:44.420786 24621 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:57:44.420851 24621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 14:57:44.420925 24672 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap starting.
I20260925 14:57:44.428985 24621 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:44033
I20260925 14:57:44.429059 24779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:44033 every 8 connection(s)
I20260925 14:57:44.429425 24621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:57:44.430706 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24621
I20260925 14:57:44.431008 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:43055
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:39829
--webserver_port=34997
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:57:44.432736 24672 log.cc:824] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:57:44.435500 24672 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:57:44.435871 24672 tablet_bootstrap.cc:492] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Bootstrap complete.
I20260925 14:57:44.437502 24672 ts_tablet_manager.cc:1397] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.013s
I20260925 14:57:44.439199 24672 raft_consensus.cc:348] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:44.439559 24672 raft_consensus.cc:729] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40498e7071e94bb0ace7c7ee3cc75d7f, State: Initialized, Role: FOLLOWER
I20260925 14:57:44.439726 24672 consensus_queue.cc:260] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:44.440023 24672 ts_tablet_manager.cc:1428] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260925 14:57:44.505280 24782 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:57:44.430182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:40961)
0925 14:57:44.430371 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.430408 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:44.431532 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:44.505043 (+ 73511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
W20260925 14:57:44.505589 24780 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:39829 (0 consecutive failures): Network error: Failed to ping master at 127.9.141.126:39829: Client connection negotiation failed: client connection to 127.9.141.126:39829: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:57:44.505944 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.505824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:35261)
0925 14:57:44.505884 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.505924 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:57:44.506273 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.506177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:38525)
0925 14:57:44.506224 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.506254 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:57:44.506398 24780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:57:44.520886 24783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.521147 24783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.521195 24783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.522575 24783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.522643 24783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.522696 24783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:57:44.524365 24783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:43055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=34997
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:44.524715 24783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.524968 24783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:44.530472 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.530493 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.530500 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.530932 24783 server_base.cc:1034] running on GCE node
I20260925 14:57:44.531158 24783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.531380 24783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.532541 24783 hybrid_clock.cc:648] HybridClock initialized: now 1790348264532521 us; error 39 us; skew 500 ppm
Sep 25 14:57:44 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.535130 24783 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:57:44.535660 24783 webserver.cc:460] Webserver started at http://127.9.141.66:34997/ using document root <none> and password file <none>
I20260925 14:57:44.535943 24783 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.536018 24783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.537691 24783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:44.538529 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.538842 24783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:44.538952 24783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "0be67d367ec24ab58418a271e15f11cb"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "e5342947b59a5343b9ab1dd0e2747cb1"
server_key_iv: "5140847d0b94f77afa7efb10dcb00608"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.539315 24783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.562024 24783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.564436 24783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.564628 24783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.564886 24783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:44.565315 24807 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 14:57:44.566900 24807 ts_tablet_manager.cc:1852] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 14:57:44.567186 24807 ts_tablet_manager.cc:1905] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:57:44.569340 24807 ts_tablet_manager.cc:1918] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:57:44.569434 24807 log.cc:1196] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/1656de6753b749588ad2cffcf6129949
I20260925 14:57:44.569573 24807 ts_tablet_manager.cc:1939] T 1656de6753b749588ad2cffcf6129949 P 0be67d367ec24ab58418a271e15f11cb: Deleting consensus metadata
I20260925 14:57:44.569767 24783 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 14:57:44.569854 24783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 14:57:44.569897 24783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:44.569924 24783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.576144 24783 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:43055
I20260925 14:57:44.576198 24914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:43055 every 8 connection(s)
I20260925 14:57:44.576517 24783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:57:44.577798 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.577182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:33729)
0925 14:57:44.577552 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.577630 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
W20260925 14:57:44.578068 24915 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:39829 (0 consecutive failures): Network error: Failed to ping master at 127.9.141.126:39829: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
I20260925 14:57:44.578384 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.578263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:42389)
0925 14:57:44.578341 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.578368 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:57:44.578800 24917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.578680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:48295)
0925 14:57:44.578747 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.578779 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 14:57:44.578979 24915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:57:44.579281 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24783
I20260925 14:57:44.579556 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:41857
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:39829
--webserver_port=44061
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:57:44.660234 24918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:57:44.660455 24918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:57:44.660480 24918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:57:44.661590 24918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:57:44.661625 24918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:57:44.661650 24918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:57:44.663021 24918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:41857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=44061
--tserver_master_addrs=127.9.141.126:39829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:57:44.663201 24918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:57:44.663367 24918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:57:44.668221 24924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:57:44.668215 24923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.668324 24918 server_base.cc:1034] running on GCE node
W20260925 14:57:44.668215 24926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:57:44.668844 24918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:57:44.669111 24918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:57:44.670311 24918 hybrid_clock.cc:648] HybridClock initialized: now 1790348264670284 us; error 47 us; skew 500 ppm
Sep 25 14:57:44 dist-test-slave-1sv7 krb5kdc[23480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:57:44.678068 24918 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:57:44.678488 24918 webserver.cc:460] Webserver started at http://127.9.141.67:44061/ using document root <none> and password file <none>
I20260925 14:57:44.678689 24918 fs_manager.cc:359] Metadata directory not provided
I20260925 14:57:44.678730 24918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:57:44.680052 24918 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:57:44.680770 24933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.680975 24918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:57:44.681064 24918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "df0b47e54bfd4eb29ff03b1079bbe798"
format_stamp: "Formatted at 2026-09-25 14:57:42 on dist-test-slave-1sv7"
server_key: "288efb788f727563513a1c4afa0d424b"
server_key_iv: "4dc0fada723047ad666174d1a13d27d4"
server_key_version: "encryptionkey@0"
I20260925 14:57:44.681402 24918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:57:44.693244 24918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:57:44.695508 24918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:57:44.695660 24918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:57:44.695887 24918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:57:44.696177 24918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:57:44.696208 24918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.696246 24918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:57:44.696273 24918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:57:44.702149 24918 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:41857
I20260925 14:57:44.702286 25046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:41857 every 8 connection(s)
I20260925 14:57:44.702579 24918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:57:44.703687 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.703298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:57105)
0925 14:57:44.703473 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.703543 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
W20260925 14:57:44.704118 25047 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:39829 (0 consecutive failures): Network error: Failed to ping master at 127.9.141.126:39829: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
I20260925 14:57:44.704509 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.704334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:44123)
0925 14:57:44.704447 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.704478 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 14:57:44.704551 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 24918
I20260925 14:57:44.704857 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:44.704745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:44663)
0925 14:57:44.704795 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:44.704821 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:57:44.704993 25047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:57:45.506996 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.506721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:58641)
0925 14:57:45.506882 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:45.506919 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260925 14:57:45.579761 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.579464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:40637)
0925 14:57:45.579683 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:45.579723 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":169,"threads_started":1}
I20260925 14:57:45.705597 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.705366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:49301)
0925 14:57:45.705505 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:45.705545 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260925 14:57:45.914677 25054 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:45.914803 25054 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:45.915118 25054 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
Sep 25 14:57:45 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.66@KRBTEST.COM
Sep 25 14:57:45 dist-test-slave-1sv7 krb5kdc[23480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348264, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.67@KRBTEST.COM
I20260925 14:57:45.926751 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.915463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:37427 (local address 127.9.141.67:41857)
0925 14:57:45.915614 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.915621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.915655 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:57:45.916216 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.916220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.916281 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.916562 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.917305 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.918680 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.919783 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.920190 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.920310 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.923134 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:45.923139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:45.923139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:45.923151 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:45.925047 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925544 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.925547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.925547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.925649 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925938 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.925939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.925939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.926202 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:45.926287 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.926509 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.926635 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260925 14:57:45.926764 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.915411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:56721 (local address 127.9.141.66:43055)
0925 14:57:45.915576 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:57:45.915590 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:57:45.915617 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:57:45.916275 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:57:45.916277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:57:45.916326 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:57:45.916477 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:57:45.917292 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.918683 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.919717 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.920127 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.920255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.922791 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:57:45.922795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:57:45.922827 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:57:45.922840 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:57:45.924716 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925218 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.925222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.925222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.925403 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925772 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:57:45.925773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:57:45.925773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:57:45.926076 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:57:45.926141 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:57:45.926495 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 14:57:45.926640 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260925 14:57:45.926769 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.915324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:43055 (local address 127.9.141.65:56721)
0925 14:57:45.915480 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:45.915490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:45.915547 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:45.916593 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:45.916594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:45.916959 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:45.917170 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:45.917178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.918814 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.918815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:45.919626 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:45.919629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.920231 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.920232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:45.920342 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.921024 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:45.921033 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:45.922662 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:45.924799 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:45.924807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:45.924809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:45.925113 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:45.925530 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:45.925531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:45.925671 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:45.926145 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:45.926151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:45.926295 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 14:57:45.926422 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:57:45.926628 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":42,"thread_start_us":78,"threads_started":1}
I20260925 14:57:45.927450 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:45.915298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:41857 (local address 127.9.141.65:37427)
0925 14:57:45.915362 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:45.915373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:57:45.915432 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:57:45.916580 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:57:45.916583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:57:45.916976 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:57:45.917168 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:45.917175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.918798 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.918799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:45.919627 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:57:45.919630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:57:45.920279 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:57:45.920281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:57:45.920343 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:57:45.921060 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:57:45.921064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:57:45.923029 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:57:45.925176 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:45.925178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:45.925407 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:45.925797 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:57:45.925798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:57:45.925798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:57:45.925861 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:57:45.926289 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:57:45.926290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:57:45.926330 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:57:45.926437 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:57:45.926705 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":192}
W20260925 14:57:45.927599 24668 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
W20260925 14:57:45.928422 24667 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
I20260925 14:57:45.928490 24667 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f; no voters: 0be67d367ec24ab58418a271e15f11cb, df0b47e54bfd4eb29ff03b1079bbe798
I20260925 14:57:45.928601 25054 raft_consensus.cc:2726] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:57:46.507762 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:46.507546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:60397)
0925 14:57:46.507674 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:46.507712 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 14:57:46.580516 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:46.580194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:54103)
0925 14:57:46.580337 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:46.580432 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260925 14:57:46.706516 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:46.706144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:37923)
0925 14:57:46.706406 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:46.706463 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":73,"threads_started":1}
I20260925 14:57:47.463692 25061 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:47.463788 25061 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:47.463940 25061 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
W20260925 14:57:47.464341 24668 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
W20260925 14:57:47.464435 24667 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
I20260925 14:57:47.464504 24668 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f; no voters: 0be67d367ec24ab58418a271e15f11cb, df0b47e54bfd4eb29ff03b1079bbe798
I20260925 14:57:47.464602 25061 raft_consensus.cc:2726] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:57:47.508589 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:47.508365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:59801)
0925 14:57:47.508499 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:47.508532 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260925 14:57:47.581182 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:47.580972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:34777)
0925 14:57:47.581105 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:47.581144 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260925 14:57:47.707165 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:47.706961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:50599)
0925 14:57:47.707084 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:47.707123 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260925 14:57:48.509683 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.509277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:50427)
0925 14:57:48.509589 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.509636 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":77,"threads_started":1}
I20260925 14:57:48.581889 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.581669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:35957)
0925 14:57:48.581807 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.581848 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260925 14:57:48.707940 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:48.707704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:50331)
0925 14:57:48.707853 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:48.707896 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260925 14:57:49.510700 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.510443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:56505)
0925 14:57:49.510602 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.510654 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260925 14:57:49.582552 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.582347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:55001)
0925 14:57:49.582472 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.582511 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260925 14:57:49.708683 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:49.708489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:48901)
0925 14:57:49.708612 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:49.708645 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260925 14:57:50.112898 25071 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:50.113015 25071 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:50.113220 25071 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
W20260925 14:57:50.113586 24668 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
W20260925 14:57:50.113664 24667 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
I20260925 14:57:50.113759 24667 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f; no voters: 0be67d367ec24ab58418a271e15f11cb, df0b47e54bfd4eb29ff03b1079bbe798
I20260925 14:57:50.113864 25071 raft_consensus.cc:2726] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:57:50.511552 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.511276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:38739)
0925 14:57:50.511457 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:50.511505 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260925 14:57:50.583397 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.583144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:35557)
0925 14:57:50.583309 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:50.583352 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260925 14:57:50.709460 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:50.709213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:58337)
0925 14:57:50.709362 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:50.709408 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260925 14:57:51.512336 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:51.512098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:38849)
0925 14:57:51.512236 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:51.512288 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 14:57:51.584177 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:51.583939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:57371)
0925 14:57:51.584103 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:51.584133 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260925 14:57:51.710300 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:51.710048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:43265)
0925 14:57:51.710212 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:51.710254 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260925 14:57:52.513190 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:52.512954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:39431)
0925 14:57:52.513099 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:52.513141 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260925 14:57:52.585029 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:52.584781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:52301)
0925 14:57:52.584899 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:52.584944 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 14:57:52.711112 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:52.710872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:41415)
0925 14:57:52.711023 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:52.711069 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260925 14:57:53.514063 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:53.513772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:36693)
0925 14:57:53.513936 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:53.513979 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260925 14:57:53.585683 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:53.585472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:58943)
0925 14:57:53.585605 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:53.585638 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260925 14:57:53.711871 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:53.711650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:33927)
0925 14:57:53.711778 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:53.711827 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260925 14:57:53.734318 25084 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:53.734431 25084 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:53.734652 25084 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
W20260925 14:57:53.735011 24668 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
W20260925 14:57:53.735097 24667 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
I20260925 14:57:53.735143 24667 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f; no voters: 0be67d367ec24ab58418a271e15f11cb, df0b47e54bfd4eb29ff03b1079bbe798
I20260925 14:57:53.735236 25084 raft_consensus.cc:2726] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:57:54.515101 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:54.514862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:39699)
0925 14:57:54.515007 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:54.515056 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260925 14:57:54.586462 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:54.586229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:60733)
0925 14:57:54.586371 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:54.586418 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260925 14:57:54.712677 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:54.712453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:42311)
0925 14:57:54.712590 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:54.712632 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260925 14:57:55.515827 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:55.515581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:33283)
0925 14:57:55.515738 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:55.515780 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260925 14:57:55.587278 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:55.587018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:49275)
0925 14:57:55.587191 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:55.587231 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260925 14:57:55.713382 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:55.713161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:38865)
0925 14:57:55.713303 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:55.713337 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260925 14:57:56.516635 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.516418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:58457)
0925 14:57:56.516552 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.516588 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260925 14:57:56.588135 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.587835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:60147)
0925 14:57:56.588043 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.588084 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260925 14:57:56.714323 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:56.714031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:56405)
0925 14:57:56.714234 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:56.714279 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":151,"threads_started":1}
I20260925 14:57:57.122540 25094 raft_consensus.cc:482] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:57:57.122638 25094 raft_consensus.cc:504] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0b47e54bfd4eb29ff03b1079bbe798" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 41857 } } peers { permanent_uuid: "0be67d367ec24ab58418a271e15f11cb" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 43055 } } peers { permanent_uuid: "40498e7071e94bb0ace7c7ee3cc75d7f" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 44033 } }
I20260925 14:57:57.122864 25094 leader_election.cc:288] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857), 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055)
W20260925 14:57:57.123260 24668 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0be67d367ec24ab58418a271e15f11cb (127.9.141.66:43055): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
W20260925 14:57:57.123407 24667 leader_election.cc:341] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df0b47e54bfd4eb29ff03b1079bbe798 (127.9.141.67:41857): Not found: Tablet not found: 1656de6753b749588ad2cffcf6129949
I20260925 14:57:57.123533 24667 leader_election.cc:302] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40498e7071e94bb0ace7c7ee3cc75d7f; no voters: 0be67d367ec24ab58418a271e15f11cb, df0b47e54bfd4eb29ff03b1079bbe798
I20260925 14:57:57.123660 25094 raft_consensus.cc:2726] T 1656de6753b749588ad2cffcf6129949 P 40498e7071e94bb0ace7c7ee3cc75d7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:57:57.517496 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.517218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:32881)
0925 14:57:57.517386 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:57.517451 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260925 14:57:57.589071 25096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.588807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:44525)
0925 14:57:57.588956 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:57.589004 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260925 14:57:57.715081 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:57.714855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:43683)
0925 14:57:57.715001 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:57.715037 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260925 14:57:58.518361 25098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.518130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:34593)
0925 14:57:58.518271 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:58.518314 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260925 14:57:58.589874 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.589625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:55027)
0925 14:57:58.589786 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:58.589829 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260925 14:57:58.715903 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:58.715628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:51261)
0925 14:57:58.715817 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:58.715859 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260925 14:57:59.519213 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.518987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.65:40373)
0925 14:57:59.519131 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.519168 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260925 14:57:59.590547 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.590349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.66:36445)
0925 14:57:59.590469 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.590504 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260925 14:57:59.713764 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24621
I20260925 14:57:59.716748 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:57:59.716551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:39829 (local address 127.9.141.67:53205)
0925 14:57:59.716669 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:57:59.716708 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:39829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260925 14:57:59.719134 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24783
I20260925 14:57:59.723625 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 24918
2026-09-25T14:57:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18690 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:57:59 dist-test-slave-1sv7 krb5kdc[25106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:57:59 dist-test-slave-1sv7 krb5kdc[25106](info): set up 2 sockets
Sep 25 14:57:59 dist-test-slave-1sv7 krb5kdc[25106](info): commencing operation
krb5kdc: starting...
W20260925 14:58:00.764539 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:58:00 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:58:00.887894 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45111
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:00.968197 25119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:00.968497 25119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:00.968538 25119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:00.969683 25119 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:58:00.969719 25119 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:58:00.969732 25119 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:58:00.969744 25119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:00.969782 25119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:00.969815 25119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:00.969831 25119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:00.969853 25119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:00.971297 25119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:00.971542 25119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:00.971778 25119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:00.976625 25127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:00.976679 25124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:00.976640 25125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:00.976815 25119 server_base.cc:1034] running on GCE node
I20260925 14:58:00.977164 25119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:00.977480 25119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:00.978657 25119 hybrid_clock.cc:648] HybridClock initialized: now 1790348280978640 us; error 74 us; skew 500 ppm
Sep 25 14:58:00 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348280, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:00.981235 25119 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:58:00.981684 25119 webserver.cc:460] Webserver started at http://127.9.141.126:43895/ using document root <none> and password file <none>
I20260925 14:58:00.981920 25119 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:00.981989 25119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:00.982163 25119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:00.983230 25119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "471ed524ed2e438580e7467f82ed072f"
format_stamp: "Formatted at 2026-09-25 14:58:00 on dist-test-slave-1sv7"
server_key: "a3cacfacb8c3162c8d36e9d981d3ae73"
server_key_iv: "6c375b2885d93c8a9d5ab09fe4de154f"
server_key_version: "encryptionkey@0"
I20260925 14:58:00.983594 25119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "471ed524ed2e438580e7467f82ed072f"
format_stamp: "Formatted at 2026-09-25 14:58:00 on dist-test-slave-1sv7"
server_key: "a3cacfacb8c3162c8d36e9d981d3ae73"
server_key_iv: "6c375b2885d93c8a9d5ab09fe4de154f"
server_key_version: "encryptionkey@0"
I20260925 14:58:00.985141 25119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:00.985966 25134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:00.986160 25119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:00.986249 25119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "471ed524ed2e438580e7467f82ed072f"
format_stamp: "Formatted at 2026-09-25 14:58:00 on dist-test-slave-1sv7"
server_key: "a3cacfacb8c3162c8d36e9d981d3ae73"
server_key_iv: "6c375b2885d93c8a9d5ab09fe4de154f"
server_key_version: "encryptionkey@0"
I20260925 14:58:00.986330 25119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:01.003031 25119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:01.005380 25119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:01.005596 25119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:01.010056 25119 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:45111
I20260925 14:58:01.010087 25186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:45111 every 8 connection(s)
I20260925 14:58:01.010485 25119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:01.011425 25187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:01.012938 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25119
I20260925 14:58:01.013065 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:58:01.013276 9781 external_mini_cluster.cc:1428] Setting key 89e0e58692e93c06a71cc3f3abf98459
I20260925 14:58:01.014418 25187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap starting.
I20260925 14:58:01.015216 25187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:01.015564 25187 log.cc:824] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:01.016296 25187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: No bootstrap required, opened a new log
I20260925 14:58:01.018784 25187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:01.018975 25187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:01.019016 25187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471ed524ed2e438580e7467f82ed072f, State: Initialized, Role: FOLLOWER
I20260925 14:58:01.019156 25187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:01.019243 25187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:01.019290 25187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:01.019347 25187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:01.020073 25187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:01.020164 25187 leader_election.cc:302] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471ed524ed2e438580e7467f82ed072f; no voters:
I20260925 14:58:01.020371 25187 leader_election.cc:288] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:01.020413 25192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:01.020524 25192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 LEADER]: Becoming Leader. State: Replica: 471ed524ed2e438580e7467f82ed072f, State: Running, Role: LEADER
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:01.020665 25192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:01.020684 25187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:01.021018 25193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "471ed524ed2e438580e7467f82ed072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } } }
I20260925 14:58:01.021035 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471ed524ed2e438580e7467f82ed072f. Latest consensus state: current_term: 1 leader_uuid: "471ed524ed2e438580e7467f82ed072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } } }
I20260925 14:58:01.021119 25193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:01.021132 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:01.022753 25208 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:01.022820 25208 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:01.022863 25198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:01.023016 25198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:01.024638 25198 catalog_manager.cc:1347] Generated new cluster ID: 704185c122e649d2b048aff4afb208f8
I20260925 14:58:01.024689 25198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:01.025023 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.013832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.9.141.126:45111)
0925 14:58:01.014009 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.014017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.014042 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:58:01.014457 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.014463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.014511 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.014669 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:01.015538 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.016571 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.017633 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.018058 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.018142 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.021213 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:01.021217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:01.021218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:01.021232 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:01.023135 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.023521 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.023524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.023525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.023657 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.023886 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.023886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.023887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.024209 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:01.024353 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.024690 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.024840 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260925 14:58:01.041819 25198 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:01.042353 25198 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:01.047672 25198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Generated new TSK 0
I20260925 14:58:01.047902 25198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:58:01.076345 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45111
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:01.157112 25213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:01.157315 25213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:01.157341 25213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:01.158432 25213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:01.158468 25213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:01.158493 25213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:58:01.159859 25213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:01.160033 25213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:01.160214 25213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:01.165627 25219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:01.165633 25218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:01.165671 25213 server_base.cc:1034] running on GCE node
W20260925 14:58:01.165647 25221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:01.166132 25213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:01.166400 25213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:01.167586 25213 hybrid_clock.cc:648] HybridClock initialized: now 1790348281167560 us; error 42 us; skew 500 ppm
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:01.170456 25213 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:58:01.170908 25213 webserver.cc:460] Webserver started at http://127.9.141.65:41243/ using document root <none> and password file <none>
I20260925 14:58:01.171089 25213 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:01.171128 25213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:01.171218 25213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:01.172242 25213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "0cad2a147da74631a20dd6796142c9ab"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "4cad0e536f71abca077a8e2e9590c1f1"
server_key_iv: "4cc306c11d57f0dba17e1f41a0a7a0ed"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.172606 25213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "0cad2a147da74631a20dd6796142c9ab"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "4cad0e536f71abca077a8e2e9590c1f1"
server_key_iv: "4cc306c11d57f0dba17e1f41a0a7a0ed"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.173832 25213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:01.174525 25228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.174700 25213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:01.174814 25213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "0cad2a147da74631a20dd6796142c9ab"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "4cad0e536f71abca077a8e2e9590c1f1"
server_key_iv: "4cc306c11d57f0dba17e1f41a0a7a0ed"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.174891 25213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:01.198798 25213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:01.201347 25213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:01.201510 25213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:01.201759 25213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:01.202085 25213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:01.202137 25213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.202183 25213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:01.202217 25213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.207744 25213 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:33369
I20260925 14:58:01.207852 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:33369 every 8 connection(s)
I20260925 14:58:01.208148 25213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:01.211186 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25213
I20260925 14:58:01.211349 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:01.211577 9781 external_mini_cluster.cc:1428] Setting key 66872479455b81e02d50a404bfbaebdb
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:01.218393 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.208920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:52703 (local address 127.9.141.126:45111)
0925 14:58:01.208992 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.208994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.209193 (+ 199us) server_negotiation.cc:407] Connection header received
0925 14:58:01.209738 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.209740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.209752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.209812 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:01.210785 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.211293 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.212734 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.212950 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.213042 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.215489 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:01.215490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:01.215491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:01.215499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:01.216965 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.217408 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.217409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.217410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.217470 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.217733 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.217734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.217735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.217850 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:01.217926 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.218204 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.218291 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:58:01.218570 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.208908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:52703)
0925 14:58:01.209058 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.209095 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.209623 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.209895 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.209898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.210467 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:01.210660 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.210665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.211418 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.211419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.212641 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.212644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.213056 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.213057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.213187 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.213760 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:01.213771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:01.215389 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:01.217074 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:01.217082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:01.217084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:01.217312 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:01.217547 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:01.217547 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:01.217548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:01.217658 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:01.217918 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:01.217925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:01.218066 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.218171 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.218327 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260925 14:58:01.219075 25342 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45111
I20260925 14:58:01.219168 25342 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:01.219368 25342 heartbeater.cc:511] Master 127.9.141.126:45111 requested a full tablet report, sending...
I20260925 14:58:01.219900 25151 ts_manager.cc:194] Registered new tserver with Master: 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369)
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
I20260925 14:58:01.220683 25151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:52703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:58:01.239496 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45111
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:01.321614 25347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:01.321844 25347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:01.321870 25347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:01.322953 25347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:01.322985 25347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:01.323010 25347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:58:01.324926 25347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:01.325100 25347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:01.325263 25347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:01.330283 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:01.330283 25355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:01.330320 25347 server_base.cc:1034] running on GCE node
W20260925 14:58:01.330274 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:01.330778 25347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:01.331022 25347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:01.332187 25347 hybrid_clock.cc:648] HybridClock initialized: now 1790348281332153 us; error 51 us; skew 500 ppm
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:01.334792 25347 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:58:01.335290 25347 webserver.cc:460] Webserver started at http://127.9.141.66:43585/ using document root <none> and password file <none>
I20260925 14:58:01.335582 25347 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:01.335656 25347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:01.335822 25347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:01.337023 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "979e31e552e5412d8f1d9819b07ed81a"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "6e15a18efc7aaaba74e5bef676b6eb56"
server_key_iv: "1c52b779238c6349b0c95065fcb624e4"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.337414 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "979e31e552e5412d8f1d9819b07ed81a"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "6e15a18efc7aaaba74e5bef676b6eb56"
server_key_iv: "1c52b779238c6349b0c95065fcb624e4"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.338788 25347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:01.339599 25362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.339774 25347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:01.339887 25347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "979e31e552e5412d8f1d9819b07ed81a"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "6e15a18efc7aaaba74e5bef676b6eb56"
server_key_iv: "1c52b779238c6349b0c95065fcb624e4"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.339965 25347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:01.355244 25347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:01.357458 25347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:01.357630 25347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:01.357890 25347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:01.358217 25347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:01.358270 25347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.358314 25347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:01.358340 25347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.364403 25347 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:34877
I20260925 14:58:01.364473 25475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:34877 every 8 connection(s)
I20260925 14:58:01.364833 25347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:01.374259 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.365635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:41723 (local address 127.9.141.126:45111)
0925 14:58:01.365698 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.365700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.365909 (+ 209us) server_negotiation.cc:407] Connection header received
0925 14:58:01.366428 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.366430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.366440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.366511 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:01.367428 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.367914 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.369000 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.369328 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.369433 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.371401 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:01.371402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:01.371402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:01.371409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:01.372862 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.373250 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.373251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.373252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.373313 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.373577 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.373578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.373578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.373702 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:01.373779 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.374073 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.374159 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:58:01.374477 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.365607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:41723)
0925 14:58:01.365767 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.365810 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.366296 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.366568 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.366571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.367142 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:01.367339 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.367344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.368006 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.368007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.368912 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.368914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.369436 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.369437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.369540 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.370006 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:01.370014 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:01.371319 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:01.372931 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:01.372937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:01.372939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:01.373143 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:01.373389 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:01.373390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:01.373391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:01.373498 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:01.373771 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:01.373778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:01.373926 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.374060 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.374240 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260925 14:58:01.374571 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25347
I20260925 14:58:01.374683 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:01.374959 9781 external_mini_cluster.cc:1428] Setting key 443f8ba4d65080905ecf94dc5c9cc17c
I20260925 14:58:01.375021 25476 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45111
I20260925 14:58:01.375124 25476 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:01.375345 25476 heartbeater.cc:511] Master 127.9.141.126:45111 requested a full tablet report, sending...
I20260925 14:58:01.375747 25151 ts_manager.cc:194] Registered new tserver with Master: 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877)
I20260925 14:58:01.376123 25151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:41723
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:58:01.401338 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45111
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:01.490108 25481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:01.490386 25481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:01.490425 25481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:01.491582 25481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:01.491649 25481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:01.491694 25481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:58:01.493275 25481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:01.493530 25481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:01.493757 25481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:01.498708 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:01.498733 25487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:01.498754 25486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:01.498929 25481 server_base.cc:1034] running on GCE node
I20260925 14:58:01.499132 25481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:01.499336 25481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:01.500474 25481 hybrid_clock.cc:648] HybridClock initialized: now 1790348281500429 us; error 60 us; skew 500 ppm
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:01.503104 25481 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:58:01.503600 25481 webserver.cc:460] Webserver started at http://127.9.141.67:42093/ using document root <none> and password file <none>
I20260925 14:58:01.503830 25481 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:01.503897 25481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:01.504053 25481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:01.505213 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "c13fbee5a8db4eda84b962094e790032"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "933cf6a7e6170f5c6dbdebf68bd04171"
server_key_iv: "e959b2941a1fd4eefb8cf8981389f820"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.505571 25481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "c13fbee5a8db4eda84b962094e790032"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "933cf6a7e6170f5c6dbdebf68bd04171"
server_key_iv: "e959b2941a1fd4eefb8cf8981389f820"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.506937 25481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:01.507709 25496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.507906 25481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:01.508011 25481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "c13fbee5a8db4eda84b962094e790032"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "933cf6a7e6170f5c6dbdebf68bd04171"
server_key_iv: "e959b2941a1fd4eefb8cf8981389f820"
server_key_version: "encryptionkey@0"
I20260925 14:58:01.508086 25481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:01.529340 25481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:01.531855 25481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:01.532035 25481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:01.532320 25481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:01.532806 25481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:01.532877 25481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.532919 25481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:01.532943 25481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:01.540900 25481 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:37447
I20260925 14:58:01.540944 25609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:37447 every 8 connection(s)
I20260925 14:58:01.541289 25481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:01.546922 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25481
I20260925 14:58:01.547050 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:01.547319 9781 external_mini_cluster.cc:1428] Setting key b916dc8dcc3d25764797c1dca1fa6b5b
Sep 25 14:58:01 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348281, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:01.551855 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.542093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:47635 (local address 127.9.141.126:45111)
0925 14:58:01.542140 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.542143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.542326 (+ 183us) server_negotiation.cc:407] Connection header received
0925 14:58:01.542877 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.542878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.542891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.542973 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:01.543932 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.544559 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.545576 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.545776 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.545859 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.548769 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:01.548770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:01.548771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:01.548778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:01.550289 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.550779 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.550781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.550781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.550848 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.551120 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.551121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.551122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.551289 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:01.551367 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.551674 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.551757 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:01.552073 25612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.542036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:47635)
0925 14:58:01.542185 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.542237 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.542747 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.543022 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.543027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.543635 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:01.543831 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.543836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.544662 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.544663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.545471 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.545473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.545871 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.545872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.545999 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.546674 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:01.546683 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:01.548664 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:01.550421 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:01.550429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:01.550432 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:01.550679 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:01.550918 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:01.550919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:01.550919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:01.551042 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:01.551359 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:01.551366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:01.551521 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.551651 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.551832 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260925 14:58:01.552608 25610 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45111
I20260925 14:58:01.552692 25610 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:01.552839 25610 heartbeater.cc:511] Master 127.9.141.126:45111 requested a full tablet report, sending...
I20260925 14:58:01.553181 25151 ts_manager.cc:194] Registered new tserver with Master: c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
I20260925 14:58:01.553586 25151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:47635
I20260925 14:58:01.559852 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:01.567967 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.561304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.9.141.126:45111)
0925 14:58:01.561366 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.561368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.561524 (+ 156us) server_negotiation.cc:407] Connection header received
0925 14:58:01.561547 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.561547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.561557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.561609 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:01.562150 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.562699 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.563468 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.563664 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.563725 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.565145 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:01.565147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:01.565147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:01.565154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:01.566496 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.566955 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.566956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.566957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.567045 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:01.567299 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:01.567300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:01.567301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:01.567565 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:01.567664 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.567779 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.567865 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:58:01.569844 25151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:01.571887 25151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:01.579114 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.574831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.9.141.66:34877)
0925 14:58:01.575001 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.575006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.575016 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:01.575051 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.575053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.575066 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.575496 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:01.575576 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.576293 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.578006 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.578627 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.578728 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.578798 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.578979 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.579023 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260925 14:58:01.579109 25624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.574595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34877 (local address 127.0.0.1:42394)
0925 14:58:01.574876 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.574886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.574923 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.575288 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.575289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.575292 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:01.575405 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.575408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.576436 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.576437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.577895 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.577898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.578740 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.578741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.578833 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.578849 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.578959 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.579026 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260925 14:58:01.579272 25621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.574470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:37447 (local address 127.0.0.1:51916)
0925 14:58:01.574687 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.574701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.574771 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.575231 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.575234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.575236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:01.575383 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.575386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.576802 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.576803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.577756 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.577758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.578932 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.578933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.578999 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.579004 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.579038 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.579088 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":67,"thread_start_us":121,"threads_started":1}
I20260925 14:58:01.579773 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.574397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33369 (local address 127.0.0.1:50674)
0925 14:58:01.574651 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.574700 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.574771 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.575277 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.575278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.575279 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:01.575395 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.575397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.576739 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.576740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.578034 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.578036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.579171 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.579171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.579595 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.579600 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.579663 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.579702 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260925 14:58:01.579869 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.574749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.9.141.67:37447)
0925 14:58:01.575051 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.575055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.575065 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:01.575099 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.575101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.575125 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.575337 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:01.575500 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.576640 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.577873 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.578816 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.579158 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.579715 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.579741 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.579781 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260925 14:58:01.580063 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.574525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.9.141.65:33369)
0925 14:58:01.575040 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.575045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.575057 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:58:01.575099 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.575101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.575117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.575197 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:01.575474 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.576625 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.578109 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.579026 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.579087 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.579138 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.579951 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.579997 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":72,"threads_started":1}
I20260925 14:58:01.580121 25410 tablet_service.cc:1467] Processing CreateTablet for tablet 392fe766da6747f58ad45e87662005ab (DEFAULT_TABLE table=test-table [id=c6557f4aec574cdfa4bbe7389ad5b8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:01.580554 25410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392fe766da6747f58ad45e87662005ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:01.580617 25544 tablet_service.cc:1467] Processing CreateTablet for tablet 392fe766da6747f58ad45e87662005ab (DEFAULT_TABLE table=test-table [id=c6557f4aec574cdfa4bbe7389ad5b8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:01.580664 25276 tablet_service.cc:1467] Processing CreateTablet for tablet 392fe766da6747f58ad45e87662005ab (DEFAULT_TABLE table=test-table [id=c6557f4aec574cdfa4bbe7389ad5b8a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:01.580958 25276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392fe766da6747f58ad45e87662005ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:01.580960 25544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 392fe766da6747f58ad45e87662005ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:01.584167 25628 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Bootstrap starting.
I20260925 14:58:01.584499 25627 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Bootstrap starting.
I20260925 14:58:01.584978 25628 tablet_bootstrap.cc:654] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:01.585137 25627 tablet_bootstrap.cc:654] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:01.585588 25628 log.cc:824] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:01.585721 25627 log.cc:824] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:01.586575 25627 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: No bootstrap required, opened a new log
I20260925 14:58:01.586648 25627 ts_tablet_manager.cc:1397] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:58:01.587946 25631 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Bootstrap starting.
I20260925 14:58:01.588737 25627 raft_consensus.cc:348] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.588897 25627 raft_consensus.cc:374] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:01.588927 25627 raft_consensus.cc:729] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979e31e552e5412d8f1d9819b07ed81a, State: Initialized, Role: FOLLOWER
I20260925 14:58:01.589077 25628 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: No bootstrap required, opened a new log
I20260925 14:58:01.589046 25627 consensus_queue.cc:260] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.589156 25628 ts_tablet_manager.cc:1397] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:58:01.589349 25476 heartbeater.cc:503] Master 127.9.141.126:45111 was elected leader, sending a full tablet report...
I20260925 14:58:01.589380 25627 ts_tablet_manager.cc:1428] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:58:01.591203 25628 raft_consensus.cc:348] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.591387 25628 raft_consensus.cc:374] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:01.591440 25628 raft_consensus.cc:729] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad2a147da74631a20dd6796142c9ab, State: Initialized, Role: FOLLOWER
I20260925 14:58:01.591571 25628 consensus_queue.cc:260] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.591840 25628 ts_tablet_manager.cc:1428] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:01.591871 25342 heartbeater.cc:503] Master 127.9.141.126:45111 was elected leader, sending a full tablet report...
I20260925 14:58:01.592060 25631 tablet_bootstrap.cc:654] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:01.592767 25631 log.cc:824] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:01.593536 25631 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: No bootstrap required, opened a new log
I20260925 14:58:01.593631 25631 ts_tablet_manager.cc:1397] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260925 14:58:01.595074 25631 raft_consensus.cc:348] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.595235 25631 raft_consensus.cc:374] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:01.595296 25631 raft_consensus.cc:729] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13fbee5a8db4eda84b962094e790032, State: Initialized, Role: FOLLOWER
I20260925 14:58:01.595433 25631 consensus_queue.cc:260] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.595723 25631 ts_tablet_manager.cc:1428] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 14:58:01.595727 25610 heartbeater.cc:503] Master 127.9.141.126:45111 was elected leader, sending a full tablet report...
I20260925 14:58:01.608587 25632 raft_consensus.cc:482] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:01.608690 25632 raft_consensus.cc:504] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.608980 25632 leader_election.cc:288] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
I20260925 14:58:01.613116 25623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.609376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:54781 (local address 127.9.141.65:33369)
0925 14:58:01.609441 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.609443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.609488 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:58:01.609583 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.609584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.609593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.609647 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:01.609835 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.610516 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.612069 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.612699 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.612811 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.612818 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.612999 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.613048 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:58:01.613116 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.609253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:41235 (local address 127.9.141.67:37447)
0925 14:58:01.609384 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.609386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.609412 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:58:01.609443 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.609444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.609455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.609521 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:01.609721 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.610514 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.612068 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.612699 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.612780 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.612790 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.612999 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.613047 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 14:58:01.613121 25478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.609149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:37447 (local address 127.9.141.66:41235)
0925 14:58:01.609212 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.609226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.609267 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.609539 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.609540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.609541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:01.609619 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.609621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.610606 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.610608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.611943 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.611946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.612802 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.612804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.612891 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.612895 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.612959 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.613021 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":54}
I20260925 14:58:01.613128 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.609149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33369 (local address 127.9.141.66:54781)
0925 14:58:01.609432 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.609441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:01.609516 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:01.609677 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:01.609678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:01.609679 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:01.609739 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.609741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.610606 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.610608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.611947 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:01.611949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.612811 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.612812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:01.612891 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.612895 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:01.612959 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:58:01.613021 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":59,"thread_start_us":183,"threads_started":1}
I20260925 14:58:01.613473 25296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392fe766da6747f58ad45e87662005ab" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad2a147da74631a20dd6796142c9ab" is_pre_election: true
I20260925 14:58:01.613480 25564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392fe766da6747f58ad45e87662005ab" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13fbee5a8db4eda84b962094e790032" is_pre_election: true
I20260925 14:58:01.613613 25564 raft_consensus.cc:2454] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 0.
I20260925 14:58:01.613626 25296 raft_consensus.cc:2454] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 0.
I20260925 14:58:01.613864 25363 leader_election.cc:302] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab, 979e31e552e5412d8f1d9819b07ed81a; no voters:
I20260925 14:58:01.614033 25632 raft_consensus.cc:2781] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:01.614109 25632 raft_consensus.cc:482] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:01.614137 25632 raft_consensus.cc:3037] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:01.614840 25632 raft_consensus.cc:504] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.614988 25632 leader_election.cc:288] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 election: Requested vote from peers 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
I20260925 14:58:01.615176 25564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392fe766da6747f58ad45e87662005ab" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13fbee5a8db4eda84b962094e790032"
I20260925 14:58:01.615200 25296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "392fe766da6747f58ad45e87662005ab" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad2a147da74631a20dd6796142c9ab"
I20260925 14:58:01.615265 25564 raft_consensus.cc:3037] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:01.615271 25296 raft_consensus.cc:3037] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Advancing to term 1
W20260925 14:58:01.615432 25477 tablet.cc:2369] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:01.616147 25296 raft_consensus.cc:2454] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 1.
I20260925 14:58:01.616147 25564 raft_consensus.cc:2454] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 1.
I20260925 14:58:01.616338 25363 leader_election.cc:302] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab, 979e31e552e5412d8f1d9819b07ed81a; no voters:
I20260925 14:58:01.616441 25632 raft_consensus.cc:2781] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:01.616575 25632 raft_consensus.cc:686] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 1 LEADER]: Becoming Leader. State: Replica: 979e31e552e5412d8f1d9819b07ed81a, State: Running, Role: LEADER
I20260925 14:58:01.616667 25632 consensus_queue.cc:237] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:01.617591 25151 catalog_manager.cc:5495] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a reported cstate change: term changed from 0 to 1, leader changed from <none> to 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66). New cstate: current_term: 1 leader_uuid: "979e31e552e5412d8f1d9819b07ed81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:01.624172 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.621500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.9.141.66:34877)
0925 14:58:01.621537 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:01.621538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:01.621547 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:01.621607 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:01.621607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:01.621621 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:01.621685 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:01.621821 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.622303 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.623227 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:01.623561 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:01.623635 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:01.623812 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:01.623933 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:01.624019 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:01.624061 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:58:01.624079 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:01.625793 25296 raft_consensus.cc:1261] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Refusing update from remote peer 979e31e552e5412d8f1d9819b07ed81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:01.625793 25564 raft_consensus.cc:1261] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Refusing update from remote peer 979e31e552e5412d8f1d9819b07ed81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:01.626128 25632 consensus_queue.cc:1035] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:01.626272 25637 consensus_queue.cc:1035] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:01.628443 25640 mvcc.cc:205] Tried to move back new op lower bound from 7333266561536704512 to 7333266561502441472. Current Snapshot: MvccSnapshot[applied={T|T < 7333266561536704512}]
I20260925 14:58:01.629141 25641 mvcc.cc:205] Tried to move back new op lower bound from 7333266561536704512 to 7333266561502441472. Current Snapshot: MvccSnapshot[applied={T|T < 7333266561536704512}]
I20260925 14:58:01.629693 25642 mvcc.cc:205] Tried to move back new op lower bound from 7333266561536704512 to 7333266561502441472. Current Snapshot: MvccSnapshot[applied={T|T < 7333266561536704512}]
I20260925 14:58:01.630859 25151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:01.630961 25151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:01.631669 25151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Sending DeleteTablet for 3 replicas of tablet 392fe766da6747f58ad45e87662005ab
I20260925 14:58:01.631965 25276 tablet_service.cc:1514] Processing DeleteTablet for tablet 392fe766da6747f58ad45e87662005ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:50674
I20260925 14:58:01.632019 25410 tablet_service.cc:1514] Processing DeleteTablet for tablet 392fe766da6747f58ad45e87662005ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:42394
I20260925 14:58:01.632016 25544 tablet_service.cc:1514] Processing DeleteTablet for tablet 392fe766da6747f58ad45e87662005ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:51916
I20260925 14:58:01.632243 25651 tablet_replica.cc:330] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: stopping tablet replica
I20260925 14:58:01.632339 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25213
I20260925 14:58:01.632351 25651 raft_consensus.cc:2229] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:01.632462 25651 raft_consensus.cc:2258] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:01.632604 25650 tablet_replica.cc:330] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: stopping tablet replica
I20260925 14:58:01.632683 25650 raft_consensus.cc:2229] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:01.632771 25650 raft_consensus.cc:2258] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:01.633165 25650 ts_tablet_manager.cc:1905] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:01.635051 25650 ts_tablet_manager.cc:1918] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:01.635128 25650 log.cc:1196] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/392fe766da6747f58ad45e87662005ab
I20260925 14:58:01.635425 25650 ts_tablet_manager.cc:1939] T 392fe766da6747f58ad45e87662005ab P c13fbee5a8db4eda84b962094e790032: Deleting consensus metadata
I20260925 14:58:01.635874 25138 catalog_manager.cc:4841] TS c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): tablet 392fe766da6747f58ad45e87662005ab (table test-table [id=c6557f4aec574cdfa4bbe7389ad5b8a0]) successfully deleted
I20260925 14:58:01.636134 25651 ts_tablet_manager.cc:1905] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:01.637777 25651 ts_tablet_manager.cc:1918] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:01.637853 25651 log.cc:1196] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/392fe766da6747f58ad45e87662005ab
I20260925 14:58:01.638126 25651 ts_tablet_manager.cc:1939] T 392fe766da6747f58ad45e87662005ab P 979e31e552e5412d8f1d9819b07ed81a: Deleting consensus metadata
I20260925 14:58:01.638661 25135 catalog_manager.cc:4841] TS 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): tablet 392fe766da6747f58ad45e87662005ab (table test-table [id=c6557f4aec574cdfa4bbe7389ad5b8a0]) successfully deleted
W20260925 14:58:01.639585 25135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:33369 (error 108)
I20260925 14:58:01.639786 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25347
I20260925 14:58:01.640445 25622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:01.639948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33369 (local address 127.0.0.1:50682)
0925 14:58:01.640353 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:01.640417 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:33369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
W20260925 14:58:01.640587 25135 catalog_manager.cc:4584] TS 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 392fe766da6747f58ad45e87662005ab: Network error: Client connection negotiation failed: client connection to 127.9.141.65:33369: connect: Connection refused (error 111)
I20260925 14:58:01.645514 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25481
I20260925 14:58:01.649869 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25119
I20260925 14:58:21.655804 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45111
--webserver_interface=127.9.141.126
--webserver_port=43895
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:21.737659 25653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:21.737872 25653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:21.737895 25653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:21.739095 25653 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:58:21.739138 25653 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:58:21.739154 25653 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:58:21.739166 25653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:21.739179 25653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:21.739224 25653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:21.739259 25653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:21.739275 25653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:21.740829 25653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=43895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:21.741081 25653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:21.741336 25653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:21.750276 25662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.750280 25660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:21.750517 25653 server_base.cc:1034] running on GCE node
W20260925 14:58:21.750276 25659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:21.751012 25653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:21.751375 25653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:21.752534 25653 hybrid_clock.cc:648] HybridClock initialized: now 1790348301752513 us; error 42 us; skew 500 ppm
Sep 25 14:58:21 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:21.761206 25653 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:58:21.761852 25653 webserver.cc:460] Webserver started at http://127.9.141.126:43895/ using document root <none> and password file <none>
I20260925 14:58:21.762130 25653 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:21.762218 25653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:21.763680 25653 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:21.764572 25669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.764855 25653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:21.764979 25653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "471ed524ed2e438580e7467f82ed072f"
format_stamp: "Formatted at 2026-09-25 14:58:00 on dist-test-slave-1sv7"
server_key: "a3cacfacb8c3162c8d36e9d981d3ae73"
server_key_iv: "6c375b2885d93c8a9d5ab09fe4de154f"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.765702 25653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:21.809719 25653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:21.812546 25653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:21.812855 25653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:21.817468 25653 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:45111
I20260925 14:58:21.817550 25721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:45111 every 8 connection(s)
I20260925 14:58:21.817960 25653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:21.819515 25722 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:58:21.820227 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap starting.
I20260925 14:58:21.821914 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25653
I20260925 14:58:21.822286 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:33369
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:45111
--webserver_port=41243
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:21.829205 25722 log.cc:824] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:21.831265 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:21.831589 25722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap complete.
I20260925 14:58:21.833977 25722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:21.834205 25722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471ed524ed2e438580e7467f82ed072f, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.834374 25722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:21.834442 25722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:21.834513 25722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:21.834566 25722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:58:21.835625 25722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:21.835722 25722 leader_election.cc:302] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471ed524ed2e438580e7467f82ed072f; no voters:
I20260925 14:58:21.835964 25722 leader_election.cc:288] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:58:21.836000 25726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:58:21.836112 25726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 LEADER]: Becoming Leader. State: Replica: 471ed524ed2e438580e7467f82ed072f, State: Running, Role: LEADER
I20260925 14:58:21.836211 25726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:21.836237 25722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:21.836540 25726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471ed524ed2e438580e7467f82ed072f. Latest consensus state: current_term: 2 leader_uuid: "471ed524ed2e438580e7467f82ed072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } } }
I20260925 14:58:21.836623 25726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:21.836546 25727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "471ed524ed2e438580e7467f82ed072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } } }
I20260925 14:58:21.836803 25727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:21.837145 25730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:21.837934 25730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:21.838068 25730 catalog_manager.cc:1259] Loaded cluster ID: 704185c122e649d2b048aff4afb208f8
I20260925 14:58:21.838342 25730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:21.839143 25730 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:21.839408 25730 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Loaded TSK: 0
I20260925 14:58:21.847965 25730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Generated new TSK 1
I20260925 14:58:21.848099 25730 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:58:21.909047 25724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:21.909286 25724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:21.909329 25724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:21.910502 25724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:21.910571 25724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:21.910617 25724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:58:21.912125 25724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:33369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=41243
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:21.912456 25724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:21.912719 25724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:21.917959 25750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.917973 25752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.917981 25749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:21.918262 25724 server_base.cc:1034] running on GCE node
I20260925 14:58:21.918515 25724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:21.918783 25724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:21.919952 25724 hybrid_clock.cc:648] HybridClock initialized: now 1790348301919934 us; error 33 us; skew 500 ppm
Sep 25 14:58:21 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:21.922842 25724 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:58:21.923393 25724 webserver.cc:460] Webserver started at http://127.9.141.65:41243/ using document root <none> and password file <none>
I20260925 14:58:21.923669 25724 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:21.923754 25724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:21.925602 25724 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:21.926424 25759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.926643 25724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:21.926733 25724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "0cad2a147da74631a20dd6796142c9ab"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "4cad0e536f71abca077a8e2e9590c1f1"
server_key_iv: "4cc306c11d57f0dba17e1f41a0a7a0ed"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.927218 25724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:21.947404 25724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:21.950939 25724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:21.951112 25724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:21.951406 25724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:21.952184 25766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:58:21.953848 25724 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:58:21.953913 25724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:58:21.953949 25724 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:58:21.954772 25724 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:58:21.954818 25724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:21.954937 25766 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Bootstrap starting.
I20260925 14:58:21.962100 25724 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:33369
I20260925 14:58:21.962337 25873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:33369 every 8 connection(s)
I20260925 14:58:21.962522 25724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:21.965883 25766 log.cc:824] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:21.968076 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25724
I20260925 14:58:21.968124 25766 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:21.968407 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:34877
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:45111
--webserver_port=43585
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:21.968453 25766 tablet_bootstrap.cc:492] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Bootstrap complete.
I20260925 14:58:21.969663 25766 ts_tablet_manager.cc:1397] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260925 14:58:21.971254 25766 raft_consensus.cc:348] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:21.971601 25766 raft_consensus.cc:729] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad2a147da74631a20dd6796142c9ab, State: Initialized, Role: FOLLOWER
I20260925 14:58:21.971783 25766 consensus_queue.cc:260] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } }
I20260925 14:58:21.972060 25766 ts_tablet_manager.cc:1428] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Sep 25 14:58:21 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:21.977831 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.963316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:53983 (local address 127.9.141.126:45111)
0925 14:58:21.963504 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.963512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.964714 (+ 1202us) server_negotiation.cc:407] Connection header received
0925 14:58:21.965642 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.965650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.965707 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.965856 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:21.966612 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.968042 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.970110 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.970719 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.970801 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.973876 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:21.973880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:21.973881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:21.973892 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:21.975936 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.976451 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.976454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.976455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.976561 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.976808 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.976808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.976809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.977073 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:21.977143 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.977487 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.977622 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260925 14:58:21.978842 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.963255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:53983)
0925 14:58:21.964526 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.964578 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.965403 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.965909 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.965913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.966245 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:21.966479 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.966485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.968292 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.968293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.970005 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.970008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.970940 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.970941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.971069 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.971791 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:21.971801 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:21.973745 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:21.976033 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.976039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.976041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.976332 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.976628 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.976628 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.976629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.976740 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.977177 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:21.977182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:21.977320 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.978465 (+ 1145us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.978642 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":109,"threads_started":1}
I20260925 14:58:21.979795 25874 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45111
I20260925 14:58:21.979892 25874 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:21.980064 25874 heartbeater.cc:511] Master 127.9.141.126:45111 requested a full tablet report, sending...
I20260925 14:58:21.980839 25686 ts_manager.cc:194] Registered new tserver with Master: 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369)
I20260925 14:58:21.982038 25686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:53983
I20260925 14:58:21.983417 25874 heartbeater.cc:503] Master 127.9.141.126:45111 was elected leader, sending a full tablet report...
I20260925 14:58:21.987816 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.982563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33369 (local address 127.0.0.1:35638)
0925 14:58:21.983065 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.983175 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.983253 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.983661 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.983665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.983667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:21.983732 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.983733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.985165 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.985166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.986508 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.986511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.987511 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.987513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.987595 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.987608 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.987683 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.987734 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":122,"threads_started":1}
I20260925 14:58:21.987828 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.982938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.9.141.65:33369)
0925 14:58:21.983381 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.983386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.983394 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:21.983427 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.983429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.983443 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.983871 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:21.983929 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.985060 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.986612 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.987420 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.987496 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.987549 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.987708 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.987755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":79,"threads_started":1}
I20260925 14:58:21.988466 25808 tablet_service.cc:1514] Processing DeleteTablet for tablet 392fe766da6747f58ad45e87662005ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:01 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:35638
I20260925 14:58:21.989082 25883 tablet_replica.cc:330] stopping tablet replica
I20260925 14:58:21.989194 25883 raft_consensus.cc:2229] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:21.989260 25883 raft_consensus.cc:2258] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:21.989557 25883 ts_tablet_manager.cc:1905] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:21.992136 25883 ts_tablet_manager.cc:1918] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:21.992218 25883 log.cc:1196] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal/wals/392fe766da6747f58ad45e87662005ab
I20260925 14:58:21.992530 25883 ts_tablet_manager.cc:1939] T 392fe766da6747f58ad45e87662005ab P 0cad2a147da74631a20dd6796142c9ab: Deleting consensus metadata
I20260925 14:58:21.993060 25670 catalog_manager.cc:4841] TS 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369): tablet 392fe766da6747f58ad45e87662005ab (table test-table [id=c6557f4aec574cdfa4bbe7389ad5b8a0]) successfully deleted
W20260925 14:58:22.054689 25879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:22.054899 25879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:22.054924 25879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:22.056102 25879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:22.056147 25879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:22.056174 25879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:58:22.057694 25879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:34877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=43585
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:22.057934 25879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:22.058180 25879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:22.063525 25879 server_base.cc:1034] running on GCE node
W20260925 14:58:22.063485 25891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.063752 25888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.063475 25889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.064013 25879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:22.064302 25879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:22.065466 25879 hybrid_clock.cc:648] HybridClock initialized: now 1790348302065445 us; error 47 us; skew 500 ppm
Sep 25 14:58:22 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:22.068094 25879 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:58:22.068595 25879 webserver.cc:460] Webserver started at http://127.9.141.66:43585/ using document root <none> and password file <none>
I20260925 14:58:22.068796 25879 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:22.068849 25879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:22.070304 25879 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:22.071045 25898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.071367 25879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:22.071462 25879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "979e31e552e5412d8f1d9819b07ed81a"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "6e15a18efc7aaaba74e5bef676b6eb56"
server_key_iv: "1c52b779238c6349b0c95065fcb624e4"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.071837 25879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:22.092274 25879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:22.094776 25879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:22.094897 25879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:22.095096 25879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:22.095361 25879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:22.095384 25879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.095407 25879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:22.095419 25879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.101102 25879 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:34877
I20260925 14:58:22.101181 26011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:34877 every 8 connection(s)
I20260925 14:58:22.101528 25879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:22.105010 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 25879
I20260925 14:58:22.105304 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:37447
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:45111
--webserver_port=42093
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:58:22 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:22.112181 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.102450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:32771 (local address 127.9.141.126:45111)
0925 14:58:22.102526 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.102529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.102844 (+ 315us) server_negotiation.cc:407] Connection header received
0925 14:58:22.103392 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.103394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.103407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.103462 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.104541 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.105049 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.106226 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.106548 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.106625 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.109208 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:22.109209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:22.109210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:22.109218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:22.110731 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.111162 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.111163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.111164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.111229 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.111498 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.111499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.111499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.111622 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:22.111699 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.111993 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.112086 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:22.112421 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.102472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:32771)
0925 14:58:22.102678 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.102744 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.103290 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.103518 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.103521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.104103 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:22.104301 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.104305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.105160 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.105161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.106124 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.106126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.106650 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.106651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.106771 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.107416 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:22.107425 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:22.109099 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:22.110837 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:22.110845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:22.110848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:22.111081 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:22.111304 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:22.111304 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:22.111305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:22.111421 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:22.111690 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:22.111696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:22.111852 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.111962 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.112129 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260925 14:58:22.112942 26012 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45111
I20260925 14:58:22.113061 26012 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:22.113324 26012 heartbeater.cc:511] Master 127.9.141.126:45111 requested a full tablet report, sending...
I20260925 14:58:22.113835 25686 ts_manager.cc:194] Registered new tserver with Master: 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877)
I20260925 14:58:22.114447 25686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:32771
W20260925 14:58:22.189081 26015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:22.189296 26015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:22.189320 26015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:22.190495 26015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:22.190538 26015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:22.190564 26015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:58:22.192077 26015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:37447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=42093
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:22.192324 26015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:22.192591 26015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:22.197592 26021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.197646 26015 server_base.cc:1034] running on GCE node
W20260925 14:58:22.197642 26023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.197613 26020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.198086 26015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:22.198289 26015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:22.199437 26015 hybrid_clock.cc:648] HybridClock initialized: now 1790348302199407 us; error 52 us; skew 500 ppm
Sep 25 14:58:22 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:22.202018 26015 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:58:22.202490 26015 webserver.cc:460] Webserver started at http://127.9.141.67:42093/ using document root <none> and password file <none>
I20260925 14:58:22.202723 26015 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:22.202781 26015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:22.204276 26015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:22.205106 26030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.205310 26015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:22.205416 26015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "c13fbee5a8db4eda84b962094e790032"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "933cf6a7e6170f5c6dbdebf68bd04171"
server_key_iv: "e959b2941a1fd4eefb8cf8981389f820"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.205837 26015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:22.221912 26015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:22.224506 26015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:22.224689 26015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:22.224939 26015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:22.225260 26015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:22.225318 26015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.225363 26015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:22.225394 26015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.231575 26015 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:37447
I20260925 14:58:22.231667 26143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:37447 every 8 connection(s)
I20260925 14:58:22.232040 26015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:22 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:22.241771 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26015
I20260925 14:58:22.242738 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.232898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:36325 (local address 127.9.141.126:45111)
0925 14:58:22.232995 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.232998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.233247 (+ 249us) server_negotiation.cc:407] Connection header received
0925 14:58:22.233733 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.233734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.233745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.233802 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.234730 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.235234 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.236581 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.236868 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.236930 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.239372 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:22.239373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:22.239374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:22.239381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:22.241038 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.241443 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.241444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.241444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.241537 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.241790 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.241792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.241792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.241961 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:22.242042 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.242491 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.242605 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:58:22.242933 26146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.232891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:36325)
0925 14:58:22.233097 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.233143 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.233627 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.233857 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.233860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.234426 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:22.234636 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.234641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.235364 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.235365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.236463 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.236466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.236936 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.236936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.237046 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.237627 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:22.237639 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:22.239272 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:22.241119 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:22.241126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:22.241128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:22.241366 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:22.241616 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:22.241617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:22.241617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:22.241715 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:22.242051 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:22.242058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:22.242243 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.242443 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.242583 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":115,"threads_started":1}
I20260925 14:58:22.243595 26144 heartbeater.cc:347] Connected to a master server at 127.9.141.126:45111
I20260925 14:58:22.243718 26144 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:22.243937 26144 heartbeater.cc:511] Master 127.9.141.126:45111 requested a full tablet report, sending...
I20260925 14:58:22.244405 25686 ts_manager.cc:194] Registered new tserver with Master: c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
I20260925 14:58:22.244807 25686 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:36325
I20260925 14:58:22.250172 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.242253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.9.141.126:45111)
0925 14:58:22.242412 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.242419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.242625 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:58:22.242707 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.242709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.242723 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.242792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.243933 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.244606 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.245643 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.245843 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.245911 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.247442 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:22.247444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:22.247444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:22.247450 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:22.249113 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.249515 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.249515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.249516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.249573 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.249746 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.249746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.249746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.249854 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:22.249987 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.250035 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.250100 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260925 14:58:22.251194 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:22.254328 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.251581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.9.141.126:45111)
0925 14:58:22.251651 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.251653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.251809 (+ 156us) server_negotiation.cc:407] Connection header received
0925 14:58:22.251900 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.251901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.251947 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.252020 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:22.252179 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.252775 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.253536 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.253752 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.253837 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.254012 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:22.254066 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 14:58:22.254260 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54536: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 14:58:22.254374 26147 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54536: token expired
W20260925 14:58:22.254436 25613 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.141.126:45111)
I20260925 14:58:22.261996 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.254719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.9.141.126:45111)
0925 14:58:22.255916 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.255918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.255937 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:58:22.255958 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.255959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.255968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.256020 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.256574 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.257032 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.257785 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.257986 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.258059 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.259274 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:22.259276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:22.259276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:22.259283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:22.260837 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.261166 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.261167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.261168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.261232 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.261440 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.261441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.261441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.261584 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:22.261689 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.261820 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.261895 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169}
I20260925 14:58:22.263283 25686 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:22.265039 25686 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:22.267649 25808 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa28cabdebb429ea13764b49bafb84c (DEFAULT_TABLE table=test-table [id=ba818b9477924692bd1d097f1feea193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:22.267869 25808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa28cabdebb429ea13764b49bafb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:22.269595 25766 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Bootstrap starting.
I20260925 14:58:22.269958 25766 tablet_bootstrap.cc:654] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:22.270596 25766 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: No bootstrap required, opened a new log
I20260925 14:58:22.270643 25766 ts_tablet_manager.cc:1397] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:22.270746 25766 raft_consensus.cc:348] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.270821 25766 raft_consensus.cc:374] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:22.270865 25766 raft_consensus.cc:729] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad2a147da74631a20dd6796142c9ab, State: Initialized, Role: FOLLOWER
I20260925 14:58:22.270918 25766 consensus_queue.cc:260] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.271020 25766 ts_tablet_manager.cc:1428] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.272043 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.267836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.9.141.67:37447)
0925 14:58:22.268031 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.268037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.268047 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:22.268087 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.268089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.268106 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.268194 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:22.268453 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.269551 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.270822 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.271639 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.271750 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.271808 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.271924 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.271960 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260925 14:58:22.272073 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.267581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.9.141.66:34877)
0925 14:58:22.267777 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.267783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.267796 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:58:22.267841 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.267844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.267862 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.267998 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:22.268118 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.269302 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.270734 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.271570 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.271697 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.271776 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.271938 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.271980 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":126,"threads_started":1}
I20260925 14:58:22.272068 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.267318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34877 (local address 127.0.0.1:55060)
0925 14:58:22.267379 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.267394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.267448 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.267995 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.267996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.267998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:22.268050 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.268052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.269411 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.269412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.270616 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.270619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.271777 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.271778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.271849 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.271853 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.271918 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.271972 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":4}
I20260925 14:58:22.272073 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.267318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:37447 (local address 127.0.0.1:58128)
0925 14:58:22.267440 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.267451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.267512 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.268236 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.268237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.268239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:22.268341 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.268342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.269743 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.269744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.270720 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.270723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.271754 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.271755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.271818 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.271822 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.271908 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.271957 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":14,"thread_start_us":60,"threads_started":1}
I20260925 14:58:22.272905 25946 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa28cabdebb429ea13764b49bafb84c (DEFAULT_TABLE table=test-table [id=ba818b9477924692bd1d097f1feea193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:22.273018 26078 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa28cabdebb429ea13764b49bafb84c (DEFAULT_TABLE table=test-table [id=ba818b9477924692bd1d097f1feea193]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:22.273274 25946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa28cabdebb429ea13764b49bafb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:22.273391 26078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa28cabdebb429ea13764b49bafb84c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:22.276286 26154 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Bootstrap starting.
I20260925 14:58:22.276568 26155 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Bootstrap starting.
I20260925 14:58:22.277006 26154 tablet_bootstrap.cc:654] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:22.277257 26155 tablet_bootstrap.cc:654] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:22.277486 26154 log.cc:824] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:22.277832 26155 log.cc:824] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:22.278200 26154 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: No bootstrap required, opened a new log
I20260925 14:58:22.278276 26154 ts_tablet_manager.cc:1397] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:22.278632 26155 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: No bootstrap required, opened a new log
I20260925 14:58:22.278761 26155 ts_tablet_manager.cc:1397] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:22.279580 26154 raft_consensus.cc:348] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.279718 26154 raft_consensus.cc:374] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:22.279742 26154 raft_consensus.cc:729] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 979e31e552e5412d8f1d9819b07ed81a, State: Initialized, Role: FOLLOWER
I20260925 14:58:22.279858 26154 consensus_queue.cc:260] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.280138 26154 ts_tablet_manager.cc:1428] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:22.280159 26012 heartbeater.cc:503] Master 127.9.141.126:45111 was elected leader, sending a full tablet report...
I20260925 14:58:22.280293 26155 raft_consensus.cc:348] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.280488 26155 raft_consensus.cc:374] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:22.280539 26155 raft_consensus.cc:729] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13fbee5a8db4eda84b962094e790032, State: Initialized, Role: FOLLOWER
I20260925 14:58:22.280833 26155 consensus_queue.cc:260] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.281122 26144 heartbeater.cc:503] Master 127.9.141.126:45111 was elected leader, sending a full tablet report...
I20260925 14:58:22.281339 26155 ts_tablet_manager.cc:1428] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:22.288352 26158 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:22.288717 26158 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.289227 26158 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447), 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369)
I20260925 14:58:22.292520 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.289457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33369 (local address 127.9.141.66:48709)
0925 14:58:22.289530 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.289544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.289592 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.289809 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.289811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.289813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:22.289909 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.289911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.290744 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.290745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.291616 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.291618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.292186 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.292187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.292276 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.292280 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.292342 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.292422 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":27}
I20260925 14:58:22.292548 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.289538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:48709 (local address 127.9.141.65:33369)
0925 14:58:22.289587 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.289589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.289672 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:58:22.289708 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.289709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.289720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.289781 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:22.289991 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.290661 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.291716 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.292095 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.292158 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.292166 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.292373 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.292457 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:22.292680 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.289538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:48177 (local address 127.9.141.67:37447)
0925 14:58:22.289595 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.289597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.289792 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:58:22.289824 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.289825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.289835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.289927 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:22.290097 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.290753 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.291745 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.292194 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.292272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.292281 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.292524 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.292574 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:22.292919 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.289457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:37447 (local address 127.9.141.66:48177)
0925 14:58:22.289669 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.289681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.289762 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.289943 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.289944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.289946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:22.290025 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.290034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.290836 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.290837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.291621 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.291623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.292297 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.292298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.292435 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.292441 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.292516 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.292555 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":26,"thread_start_us":121,"threads_started":1}
I20260925 14:58:22.292893 25828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa28cabdebb429ea13764b49bafb84c" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad2a147da74631a20dd6796142c9ab" is_pre_election: true
I20260925 14:58:22.293098 25828 raft_consensus.cc:2454] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 0.
I20260925 14:58:22.293277 26098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa28cabdebb429ea13764b49bafb84c" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13fbee5a8db4eda84b962094e790032" is_pre_election: true
I20260925 14:58:22.293360 25899 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab, 979e31e552e5412d8f1d9819b07ed81a; no voters:
I20260925 14:58:22.293443 26098 raft_consensus.cc:2454] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 0.
I20260925 14:58:22.293488 26158 raft_consensus.cc:2781] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:22.293545 26158 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:22.293569 26158 raft_consensus.cc:3037] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:22.294337 26158 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.294493 26158 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 election: Requested vote from peers c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447), 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369)
I20260925 14:58:22.294652 26098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa28cabdebb429ea13764b49bafb84c" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13fbee5a8db4eda84b962094e790032"
I20260925 14:58:22.294655 25828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa28cabdebb429ea13764b49bafb84c" candidate_uuid: "979e31e552e5412d8f1d9819b07ed81a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cad2a147da74631a20dd6796142c9ab"
I20260925 14:58:22.294750 25828 raft_consensus.cc:3037] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:22.294754 26098 raft_consensus.cc:3037] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:22.295542 26098 raft_consensus.cc:2454] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 1.
I20260925 14:58:22.295766 25902 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032; no voters:
I20260925 14:58:22.295872 26158 raft_consensus.cc:2781] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:22.296058 26158 raft_consensus.cc:686] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 1 LEADER]: Becoming Leader. State: Replica: 979e31e552e5412d8f1d9819b07ed81a, State: Running, Role: LEADER
I20260925 14:58:22.296099 25828 raft_consensus.cc:2454] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 979e31e552e5412d8f1d9819b07ed81a in term 1.
I20260925 14:58:22.296180 26158 consensus_queue.cc:237] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:22.297029 25685 catalog_manager.cc:5495] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a reported cstate change: term changed from 0 to 1, leader changed from <none> to 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66). New cstate: current_term: 1 leader_uuid: "979e31e552e5412d8f1d9819b07ed81a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:22.309476 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.306577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.9.141.66:34877)
0925 14:58:22.306661 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.306664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.306673 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:22.306716 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.306717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.306733 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.306815 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:22.306983 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.307496 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.308488 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.308703 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.308766 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.309059 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:22.309163 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:22.309236 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.309372 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.309394 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:58:22.311631 25828 raft_consensus.cc:1261] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Refusing update from remote peer 979e31e552e5412d8f1d9819b07ed81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:22.311784 26098 raft_consensus.cc:1261] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Refusing update from remote peer 979e31e552e5412d8f1d9819b07ed81a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:22.312065 26161 consensus_queue.cc:1035] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:22.312271 26158 consensus_queue.cc:1035] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:22.314972 26163 mvcc.cc:205] Tried to move back new op lower bound from 7333266646264909824 to 7333266646205972480. Current Snapshot: MvccSnapshot[applied={T|T < 7333266646264909824}]
I20260925 14:58:22.315236 26164 mvcc.cc:205] Tried to move back new op lower bound from 7333266646264909824 to 7333266646205972480. Current Snapshot: MvccSnapshot[applied={T|T < 7333266646264909824}]
I20260925 14:58:22.315831 26165 mvcc.cc:205] Tried to move back new op lower bound from 7333266646264909824 to 7333266646205972480. Current Snapshot: MvccSnapshot[applied={T|T < 7333266646264909824}]
I20260925 14:58:22.317513 25685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:22.317600 25685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:22.318373 25685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Sending DeleteTablet for 3 replicas of tablet 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:22.318671 25946 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa28cabdebb429ea13764b49bafb84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:22 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:55060
I20260925 14:58:22.318753 26078 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa28cabdebb429ea13764b49bafb84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:22 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:58128
I20260925 14:58:22.318797 25808 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa28cabdebb429ea13764b49bafb84c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:22 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:35638
I20260925 14:58:22.318895 25883 tablet_replica.cc:330] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: stopping tablet replica
I20260925 14:58:22.318943 25883 raft_consensus.cc:2229] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:22.318974 26172 tablet_replica.cc:330] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: stopping tablet replica
I20260925 14:58:22.318998 25883 raft_consensus.cc:2258] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:22.319056 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25724
I20260925 14:58:22.319069 26172 raft_consensus.cc:2229] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:22.319159 26172 raft_consensus.cc:2258] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:22.320218 26172 ts_tablet_manager.cc:1905] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:22.321525 26173 tablet_replica.cc:330] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: stopping tablet replica
I20260925 14:58:22.321717 26173 raft_consensus.cc:2229] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:22.321903 26173 raft_consensus.cc:2258] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:22.322445 26173 ts_tablet_manager.cc:1905] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:22.322572 26172 ts_tablet_manager.cc:1918] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:22.322664 26172 log.cc:1196] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:22.322949 26172 ts_tablet_manager.cc:1939] T 3fa28cabdebb429ea13764b49bafb84c P 979e31e552e5412d8f1d9819b07ed81a: Deleting consensus metadata
I20260925 14:58:22.323585 25670 catalog_manager.cc:4841] TS 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): tablet 3fa28cabdebb429ea13764b49bafb84c (table test-table [id=ba818b9477924692bd1d097f1feea193]) successfully deleted
I20260925 14:58:22.324817 26173 ts_tablet_manager.cc:1918] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:22.324963 26173 log.cc:1196] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:22.325327 26173 ts_tablet_manager.cc:1939] T 3fa28cabdebb429ea13764b49bafb84c P c13fbee5a8db4eda84b962094e790032: Deleting consensus metadata
W20260925 14:58:22.325855 25670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:33369 (error 108)
I20260925 14:58:22.326061 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25879
I20260925 14:58:22.325865 25673 catalog_manager.cc:4841] TS c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): tablet 3fa28cabdebb429ea13764b49bafb84c (table test-table [id=ba818b9477924692bd1d097f1feea193]) successfully deleted
I20260925 14:58:22.327292 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.327133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33369 (local address 127.0.0.1:35640)
0925 14:58:22.327214 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.327262 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:33369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 14:58:22.327515 25670 catalog_manager.cc:4584] TS 0cad2a147da74631a20dd6796142c9ab (127.9.141.65:33369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fa28cabdebb429ea13764b49bafb84c: Network error: Client connection negotiation failed: client connection to 127.9.141.65:33369: connect: Connection refused (error 111)
I20260925 14:58:22.332168 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26015
I20260925 14:58:22.338281 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 25653
I20260925 14:58:42.344923 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45111
--webserver_interface=127.9.141.126
--webserver_port=43895
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:42.427413 26175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:42.427644 26175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:42.427668 26175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:42.429010 26175 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:58:42.429055 26175 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:58:42.429070 26175 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:58:42.429083 26175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:42.429097 26175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:42.429158 26175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:42.429178 26175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:42.429196 26175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:42.430822 26175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=43895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:42.431097 26175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:42.431360 26175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:42.436661 26181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.436658 26180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.436647 26183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.436956 26175 server_base.cc:1034] running on GCE node
I20260925 14:58:42.437211 26175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:42.437500 26175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:42.438684 26175 hybrid_clock.cc:648] HybridClock initialized: now 1790348322438666 us; error 41 us; skew 500 ppm
Sep 25 14:58:42 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:42.445142 26175 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:58:42.445688 26175 webserver.cc:460] Webserver started at http://127.9.141.126:43895/ using document root <none> and password file <none>
I20260925 14:58:42.445922 26175 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:42.445971 26175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:42.447350 26175 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.448109 26190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.448314 26175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.448437 26175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "471ed524ed2e438580e7467f82ed072f"
format_stamp: "Formatted at 2026-09-25 14:58:00 on dist-test-slave-1sv7"
server_key: "a3cacfacb8c3162c8d36e9d981d3ae73"
server_key_iv: "6c375b2885d93c8a9d5ab09fe4de154f"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.448819 26175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:42.459024 26175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:42.461396 26175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:42.461508 26175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:42.465807 26175 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:45111
I20260925 14:58:42.466156 26242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:45111 every 8 connection(s)
I20260925 14:58:42.466225 26175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:42.467459 26243 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:58:42.468173 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap starting.
I20260925 14:58:42.470147 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26175
I20260925 14:58:42.470500 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:33369
--local_ip_for_outbound_sockets=127.9.141.65
--tserver_master_addrs=127.9.141.126:45111
--webserver_port=41243
--webserver_interface=127.9.141.65
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:42.477607 26243 log.cc:824] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:42.479851 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:42.480194 26243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Bootstrap complete.
I20260925 14:58:42.482496 26243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:42.482725 26243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471ed524ed2e438580e7467f82ed072f, State: Initialized, Role: FOLLOWER
I20260925 14:58:42.482892 26243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:42.482962 26243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:42.483026 26243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:42.483076 26243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:58:42.484143 26243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:42.484246 26243 leader_election.cc:302] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471ed524ed2e438580e7467f82ed072f; no voters:
I20260925 14:58:42.484460 26243 leader_election.cc:288] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:58:42.484503 26247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:58:42.484632 26247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [term 3 LEADER]: Becoming Leader. State: Replica: 471ed524ed2e438580e7467f82ed072f, State: Running, Role: LEADER
I20260925 14:58:42.484743 26243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:42.484720 26247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } }
I20260925 14:58:42.485010 26247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471ed524ed2e438580e7467f82ed072f. Latest consensus state: current_term: 3 leader_uuid: "471ed524ed2e438580e7467f82ed072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } } }
I20260925 14:58:42.485081 26247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:42.485250 26248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "471ed524ed2e438580e7467f82ed072f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471ed524ed2e438580e7467f82ed072f" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45111 } } }
I20260925 14:58:42.485328 26248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:42.485440 26251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:42.486325 26251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:42.486449 26251 catalog_manager.cc:1259] Loaded cluster ID: 704185c122e649d2b048aff4afb208f8
I20260925 14:58:42.486493 26251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:42.487426 26251 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:42.487627 26251 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Loaded TSK: 0
I20260925 14:58:42.487690 26251 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 471ed524ed2e438580e7467f82ed072f: Loaded TSK: 1
F20260925 14:58:42.487911 26251 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f87803a29ad google::LogMessage::Fail() at ??:0
@ 0x7f87803a588a google::LogMessage::SendToLog() at ??:0
@ 0x7f87803a24de google::LogMessage::Flush() at ??:0
@ 0x7f87803a3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8782b0f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8782b0cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f87882fc1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f87882fc48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f87882fc594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f87882c2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f87882ccad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f87822700d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8782265954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f87830c96db start_thread at ??:0
@ 0x7f877eea061f clone at ??:0
W20260925 14:58:42.569366 26245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:42.569612 26245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:42.569664 26245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:42.570878 26245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:42.570947 26245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:42.570997 26245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:58:42.572588 26245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:33369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=41243
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:42.572860 26245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:42.573099 26245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:42.578598 26284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.578616 26245 server_base.cc:1034] running on GCE node
W20260925 14:58:42.578567 26285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.578657 26287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.579017 26245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:42.579355 26245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:42.580502 26245 hybrid_clock.cc:648] HybridClock initialized: now 1790348322580480 us; error 38 us; skew 500 ppm
Sep 25 14:58:42 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:42.583313 26245 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:58:42.583809 26245 webserver.cc:460] Webserver started at http://127.9.141.65:41243/ using document root <none> and password file <none>
I20260925 14:58:42.584070 26245 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:42.584141 26245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:42.585690 26245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.586444 26294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.586632 26245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.586776 26245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "0cad2a147da74631a20dd6796142c9ab"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "4cad0e536f71abca077a8e2e9590c1f1"
server_key_iv: "4cc306c11d57f0dba17e1f41a0a7a0ed"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.587148 26245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:42.603082 26245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:42.605535 26245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:42.605717 26245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:42.605988 26245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:42.606460 26301 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:58:42.608561 26245 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:58:42.608624 26245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:58:42.608682 26245 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:58:42.609302 26245 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:58:42.609359 26245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:42.609436 26301 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Bootstrap starting.
I20260925 14:58:42.616675 26245 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:33369
I20260925 14:58:42.616807 26408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:33369 every 8 connection(s)
I20260925 14:58:42.617115 26245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:42.620870 26301 log.cc:824] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:42.623795 26301 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:42.624151 26301 tablet_bootstrap.cc:492] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Bootstrap complete.
I20260925 14:58:42.625452 26301 ts_tablet_manager.cc:1397] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260925 14:58:42.626258 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26245
I20260925 14:58:42.626616 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:34877
--local_ip_for_outbound_sockets=127.9.141.66
--tserver_master_addrs=127.9.141.126:45111
--webserver_port=43585
--webserver_interface=127.9.141.66
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:42.626961 26301 raft_consensus.cc:348] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:42.627507 26301 raft_consensus.cc:729] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad2a147da74631a20dd6796142c9ab, State: Initialized, Role: FOLLOWER
I20260925 14:58:42.627679 26301 consensus_queue.cc:260] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:42.628021 26301 ts_tablet_manager.cc:1428] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
W20260925 14:58:42.724114 26411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:42.617834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:54009)
0925 14:58:42.618018 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.618064 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:42.618540 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:42.723900 (+105360us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
W20260925 14:58:42.724429 26409 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:45111 (0 consecutive failures): Network error: Failed to ping master at 127.9.141.126:45111: Client connection negotiation failed: client connection to 127.9.141.126:45111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:58:42.724784 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.724669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:55331)
0925 14:58:42.724722 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.724765 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:58:42.725172 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.725086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:50877)
0925 14:58:42.725129 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.725157 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:58:42.725304 26409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:58:42.734747 26413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:42.734949 26413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:42.734973 26413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:42.736158 26413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:42.736202 26413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:42.736229 26413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:58:42.737763 26413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:34877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=43585
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:42.738008 26413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:42.738247 26413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:42.743324 26419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.743320 26422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.743609 26413 server_base.cc:1034] running on GCE node
W20260925 14:58:42.743327 26420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.744908 26413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:42.745208 26413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:42.746403 26413 hybrid_clock.cc:648] HybridClock initialized: now 1790348322746392 us; error 49 us; skew 500 ppm
Sep 25 14:58:42 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:42.749106 26413 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:58:42.749518 26413 webserver.cc:460] Webserver started at http://127.9.141.66:43585/ using document root <none> and password file <none>
I20260925 14:58:42.749706 26413 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:42.749747 26413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:42.751039 26413 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.751590 26429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.751847 26413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.751976 26413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "979e31e552e5412d8f1d9819b07ed81a"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "6e15a18efc7aaaba74e5bef676b6eb56"
server_key_iv: "1c52b779238c6349b0c95065fcb624e4"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.752411 26413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:42.764948 26413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:42.766996 26413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:42.767110 26413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:42.767380 26413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:42.767689 26413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:42.767729 26413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.767752 26413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:42.767796 26413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.773746 26413 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:34877
I20260925 14:58:42.773828 26542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:34877 every 8 connection(s)
I20260925 14:58:42.774166 26413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:42.775390 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.774966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:59233)
0925 14:58:42.775171 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.775245 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
W20260925 14:58:42.775832 26543 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:45111 (0 consecutive failures): Network error: Failed to ping master at 127.9.141.126:45111: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
I20260925 14:58:42.776145 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.776040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:58627)
0925 14:58:42.776087 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.776115 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:58:42.776511 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.776457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:34725)
0925 14:58:42.776480 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.776499 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 14:58:42.776633 26543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:58:42.783221 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26413
I20260925 14:58:42.783529 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:37447
--local_ip_for_outbound_sockets=127.9.141.67
--tserver_master_addrs=127.9.141.126:45111
--webserver_port=42093
--webserver_interface=127.9.141.67
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:42.865855 26546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:42.866061 26546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:42.866086 26546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:42.867292 26546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:42.867342 26546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:42.867370 26546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:58:42.869549 26546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:37447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=42093
--tserver_master_addrs=127.9.141.126:45111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:42.869818 26546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:42.870064 26546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:42.875641 26551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.875718 26552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:42.875800 26554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:42.875789 26546 server_base.cc:1034] running on GCE node
I20260925 14:58:42.876045 26546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:42.876304 26546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:42.877480 26546 hybrid_clock.cc:648] HybridClock initialized: now 1790348322877464 us; error 30 us; skew 500 ppm
Sep 25 14:58:42 dist-test-slave-1sv7 krb5kdc[25106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:42.880509 26546 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:58:42.881028 26546 webserver.cc:460] Webserver started at http://127.9.141.67:42093/ using document root <none> and password file <none>
I20260925 14:58:42.881304 26546 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:42.881385 26546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:42.883050 26546 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:42.883818 26561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.884155 26546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:42.884262 26546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "c13fbee5a8db4eda84b962094e790032"
format_stamp: "Formatted at 2026-09-25 14:58:01 on dist-test-slave-1sv7"
server_key: "933cf6a7e6170f5c6dbdebf68bd04171"
server_key_iv: "e959b2941a1fd4eefb8cf8981389f820"
server_key_version: "encryptionkey@0"
I20260925 14:58:42.884696 26546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:42.904778 26546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:42.907182 26546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:42.907392 26546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:42.907660 26546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:42.907991 26546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:42.908051 26546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.908087 26546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:42.908135 26546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:42.914242 26546 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:37447
I20260925 14:58:42.914297 26674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:37447 every 8 connection(s)
I20260925 14:58:42.914628 26546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:42.915673 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.915271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:57711)
0925 14:58:42.915424 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.915533 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260925 14:58:42.916224 26675 heartbeater.cc:644] Failed to heartbeat to 127.9.141.126:45111 (0 consecutive failures): Network error: Failed to ping master at 127.9.141.126:45111: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
I20260925 14:58:42.916613 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.916487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:34443)
0925 14:58:42.916560 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.916591 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:58:42.916914 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.916859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:56251)
0925 14:58:42.916880 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.916899 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260925 14:58:42.917054 26675 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:58:42.919117 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26546
I20260925 14:58:43.726087 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.725838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:33883)
0925 14:58:43.725957 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.725995 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260925 14:58:43.777153 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.776949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:54497)
0925 14:58:43.777059 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.777095 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260925 14:58:43.917647 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.917405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:47687)
0925 14:58:43.917553 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.917589 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260925 14:58:44.377375 26682 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:44.377513 26682 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:44.377985 26682 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
Sep 25 14:58:44 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.66@KRBTEST.COM
Sep 25 14:58:44 dist-test-slave-1sv7 krb5kdc[25106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348322, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.67@KRBTEST.COM
I20260925 14:58:44.391252 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.378276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:52647 (local address 127.9.141.66:34877)
0925 14:58:44.378446 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.378459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.378486 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:58:44.379086 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.379089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.379138 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.379297 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.380311 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.381770 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.383406 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.384047 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.384166 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.386977 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.386981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.386981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.387004 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.389426 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.389968 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.389971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.389972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.390112 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.390364 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.390365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.390366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.390712 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.390823 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.391009 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.391140 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":57,"threads_started":1}
I20260925 14:58:44.391261 26686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.378325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:52561 (local address 127.9.141.67:37447)
0925 14:58:44.378523 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.378530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.378556 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:58:44.379075 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.379078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.379138 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.379297 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.380311 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.381769 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.383406 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.384047 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.384163 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.387332 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:44.387336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:44.387336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:44.387353 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:44.389509 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.389962 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.389965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.389965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.390092 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:44.390363 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:44.390363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:44.390364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:44.390615 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:44.390707 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:44.391002 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 14:58:44.391131 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260925 14:58:44.391258 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.378117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34877 (local address 127.9.141.65:52647)
0925 14:58:44.378319 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.378330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.378395 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.379377 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.379381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.379900 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.380169 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.380177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.381997 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.381998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.383309 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.383312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.384169 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.384170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.384312 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.385014 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.385026 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.386865 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:44.389508 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.389515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.389517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.389860 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.390173 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.390174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.390174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.390288 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.390806 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:44.390807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:44.390855 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.390943 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.391162 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":64,"thread_start_us":50,"threads_started":1}
I20260925 14:58:44.391445 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.378151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:37447 (local address 127.9.141.65:52561)
0925 14:58:44.378343 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.378355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.378416 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.379377 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.379381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.379902 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.380169 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.380177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.381999 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.382000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.383309 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.383312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.384168 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.384170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.384312 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.385037 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.385041 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.387227 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:44.389609 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.389609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.389610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.389864 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.390165 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:44.390165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:44.390166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:44.390286 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:44.390688 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:44.390695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:44.390833 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:58:44.390941 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:58:44.391243 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":207,"thread_start_us":109,"threads_started":1}
W20260925 14:58:44.392566 26295 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
W20260925 14:58:44.392659 26298 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:44.392694 26295 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab; no voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032
I20260925 14:58:44.392810 26682 raft_consensus.cc:2726] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:58:44.726790 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.726649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:60383)
0925 14:58:44.726709 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.726751 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:58:44.777953 26687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.777634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:36545)
0925 14:58:44.777851 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.777908 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
I20260925 14:58:44.918491 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.918246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:43497)
0925 14:58:44.918403 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.918450 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260925 14:58:45.727711 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.727433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:41523)
0925 14:58:45.727619 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:45.727666 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260925 14:58:45.778636 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.778429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:43201)
0925 14:58:45.778557 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:45.778597 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260925 14:58:45.919358 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.919072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:49585)
0925 14:58:45.919263 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:45.919311 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":114,"threads_started":1}
I20260925 14:58:46.541764 26692 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:46.541900 26692 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:46.542228 26692 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
W20260925 14:58:46.542616 26298 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
W20260925 14:58:46.542701 26295 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:46.542814 26295 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab; no voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032
I20260925 14:58:46.542930 26692 raft_consensus.cc:2726] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:58:46.728652 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.728350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:51851)
0925 14:58:46.728559 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.728604 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":139,"threads_started":1}
I20260925 14:58:46.779474 26694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.779238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:42961)
0925 14:58:46.779375 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.779412 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260925 14:58:46.920195 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.919931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:50171)
0925 14:58:46.920093 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.920136 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260925 14:58:47.729559 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.729271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:41067)
0925 14:58:47.729467 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.729513 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
I20260925 14:58:47.780148 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.779925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:33967)
0925 14:58:47.780062 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.780102 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260925 14:58:47.920951 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.920738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:60423)
0925 14:58:47.920872 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.920904 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260925 14:58:48.730394 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.730152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:43063)
0925 14:58:48.730304 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.730347 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260925 14:58:48.780907 26700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.780651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:39157)
0925 14:58:48.780812 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.780854 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260925 14:58:48.921875 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.921622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:33669)
0925 14:58:48.921781 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.921825 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260925 14:58:49.055538 26702 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:49.055670 26702 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:49.055962 26702 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
W20260925 14:58:49.056303 26298 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
W20260925 14:58:49.056413 26295 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:49.056448 26295 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab; no voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032
I20260925 14:58:49.056559 26702 raft_consensus.cc:2726] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:58:49.731277 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.731047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:39399)
0925 14:58:49.731195 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.731231 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260925 14:58:49.781723 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.781464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:39203)
0925 14:58:49.781631 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.781676 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260925 14:58:49.922719 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.922454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:59737)
0925 14:58:49.922636 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.922672 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260925 14:58:50.732182 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.731926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:50983)
0925 14:58:50.732085 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.732133 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260925 14:58:50.782552 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.782301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:53137)
0925 14:58:50.782451 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.782489 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260925 14:58:50.923573 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.923258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:42077)
0925 14:58:50.923479 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.923524 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260925 14:58:51.557602 26709 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:51.557708 26709 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:51.557893 26709 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
W20260925 14:58:51.558310 26295 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
W20260925 14:58:51.558414 26298 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:51.558457 26298 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab; no voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032
I20260925 14:58:51.558568 26709 raft_consensus.cc:2726] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:58:51.733059 26710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:51.732796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:52399)
0925 14:58:51.732966 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:51.733011 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260925 14:58:51.783365 26711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:51.783139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:48159)
0925 14:58:51.783280 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:51.783319 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260925 14:58:51.924455 26712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:51.924144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:60733)
0925 14:58:51.924306 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:51.924351 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260925 14:58:52.733953 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:52.733693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:35101)
0925 14:58:52.733842 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:52.733892 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260925 14:58:52.784240 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:52.784003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:45767)
0925 14:58:52.784148 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:52.784191 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260925 14:58:52.925233 26715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:52.924986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:42783)
0925 14:58:52.925155 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:52.925186 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260925 14:58:53.734882 26716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.734595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:46629)
0925 14:58:53.734757 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:53.734831 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260925 14:58:53.785020 26717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.784795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:39755)
0925 14:58:53.784939 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:53.784974 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260925 14:58:53.926087 26718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.925847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:53315)
0925 14:58:53.925983 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:53.926032 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260925 14:58:54.518594 26719 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:54.518714 26719 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:54.518920 26719 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
W20260925 14:58:54.519353 26298 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
W20260925 14:58:54.519472 26295 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:54.519537 26295 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab; no voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032
I20260925 14:58:54.519632 26719 raft_consensus.cc:2726] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:58:54.735968 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.735675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:38217)
0925 14:58:54.735857 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.735908 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260925 14:58:54.785840 26721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.785580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:38379)
0925 14:58:54.785746 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.785791 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260925 14:58:54.926869 26722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.926645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:44393)
0925 14:58:54.926772 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.926813 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260925 14:58:55.737011 26723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.736638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:56485)
0925 14:58:55.736891 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:55.736940 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":117,"threads_started":1}
I20260925 14:58:55.786744 26724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.786467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:53703)
0925 14:58:55.786628 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:55.786675 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260925 14:58:55.927722 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.927459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:35259)
0925 14:58:55.927625 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:55.927673 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260925 14:58:56.224624 26726 raft_consensus.cc:482] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:56.224723 26726 raft_consensus.cc:504] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "979e31e552e5412d8f1d9819b07ed81a" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34877 } } peers { permanent_uuid: "c13fbee5a8db4eda84b962094e790032" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 37447 } } peers { permanent_uuid: "0cad2a147da74631a20dd6796142c9ab" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33369 } }
I20260925 14:58:56.224967 26726 leader_election.cc:288] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877), c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447)
W20260925 14:58:56.225461 26298 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c13fbee5a8db4eda84b962094e790032 (127.9.141.67:37447): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
W20260925 14:58:56.225555 26295 leader_election.cc:341] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 979e31e552e5412d8f1d9819b07ed81a (127.9.141.66:34877): Not found: Tablet not found: 3fa28cabdebb429ea13764b49bafb84c
I20260925 14:58:56.225606 26295 leader_election.cc:302] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cad2a147da74631a20dd6796142c9ab; no voters: 979e31e552e5412d8f1d9819b07ed81a, c13fbee5a8db4eda84b962094e790032
I20260925 14:58:56.225708 26726 raft_consensus.cc:2726] T 3fa28cabdebb429ea13764b49bafb84c P 0cad2a147da74631a20dd6796142c9ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:58:56.737762 26727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:56.737531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:41535)
0925 14:58:56.737672 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:56.737713 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260925 14:58:56.787662 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:56.787358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:40965)
0925 14:58:56.787559 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:56.787614 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":62,"threads_started":1}
I20260925 14:58:56.928506 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:56.928246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:35451)
0925 14:58:56.928416 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:56.928452 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260925 14:58:57.738692 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.738457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.65:60941)
0925 14:58:57.738600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.738646 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260925 14:58:57.788645 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.788353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.66:60887)
0925 14:58:57.788553 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.788593 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":147,"threads_started":1}
I20260925 14:58:57.927376 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26245
I20260925 14:58:57.929379 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.929112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:45111 (local address 127.9.141.67:46227)
0925 14:58:57.929251 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.929299 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.126:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":52,"threads_started":1}
I20260925 14:58:57.932359 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26413
I20260925 14:58:57.936869 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26546
2026-09-25T14:58:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58213 ms)
[----------] 30 tests from SecurityITest (232581 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:58 dist-test-slave-1sv7 krb5kdc[26735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:58 dist-test-slave-1sv7 krb5kdc[26735](info): set up 2 sockets
Sep 25 14:58:58 dist-test-slave-1sv7 krb5kdc[26735](info): commencing operation
krb5kdc: starting...
W20260925 14:58:59.036227 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:58:59.160130 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46037
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:46037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.243561 26748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.243795 26748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.243820 26748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.245169 26748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:59.245221 26748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.245235 26748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.245249 26748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:59.245261 26748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:59.246752 26748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:46037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:59.246958 26748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.247143 26748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.252562 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.252628 26748 server_base.cc:1034] running on GCE node
W20260925 14:58:59.252576 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.252573 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.253140 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.253453 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.254643 26748 hybrid_clock.cc:648] HybridClock initialized: now 1790348339254627 us; error 56 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.257294 26748 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:58:59.257822 26748 webserver.cc:460] Webserver started at http://127.9.141.126:43029/ using document root <none> and password file <none>
I20260925 14:58:59.258061 26748 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.258131 26748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.258282 26748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.259352 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "32529958fdbd4620baf8acf1574269be"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "2403cbec0eca0b77149626cf29adc0e4"
server_key_iv: "124c537546c85e585e18d62f69222c2f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.259725 26748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "32529958fdbd4620baf8acf1574269be"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "2403cbec0eca0b77149626cf29adc0e4"
server_key_iv: "124c537546c85e585e18d62f69222c2f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.261277 26748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:59.262087 26763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.262362 26748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:59.262463 26748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "32529958fdbd4620baf8acf1574269be"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "2403cbec0eca0b77149626cf29adc0e4"
server_key_iv: "124c537546c85e585e18d62f69222c2f"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.262521 26748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.270494 26748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.273000 26748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.273132 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.277029 26748 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:46037
I20260925 14:58:59.277078 26815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:46037 every 8 connection(s)
I20260925 14:58:59.277465 26748 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:59.278501 26816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.281179 26816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be: Bootstrap starting.
I20260925 14:58:59.281824 26816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.282131 26816 log.cc:824] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.282912 26816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be: No bootstrap required, opened a new log
I20260925 14:58:59.284590 26816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32529958fdbd4620baf8acf1574269be" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46037 } }
I20260925 14:58:59.284741 26816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.284765 26816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32529958fdbd4620baf8acf1574269be, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.284930 26816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32529958fdbd4620baf8acf1574269be" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46037 } }
I20260925 14:58:59.285008 26816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:59.285051 26816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:59.285104 26816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.285413 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26748
I20260925 14:58:59.285502 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:58:59.285732 9781 external_mini_cluster.cc:1428] Setting key 0e29e1c624e0215d3ebc0ce50387eace
I20260925 14:58:59.285833 26816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32529958fdbd4620baf8acf1574269be" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46037 } }
I20260925 14:58:59.285971 26816 leader_election.cc:302] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32529958fdbd4620baf8acf1574269be; no voters:
I20260925 14:58:59.286149 26816 leader_election.cc:288] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:59.286266 26819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:59.286489 26816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:59.286541 26819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [term 1 LEADER]: Becoming Leader. State: Replica: 32529958fdbd4620baf8acf1574269be, State: Running, Role: LEADER
I20260925 14:58:59.286684 26819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32529958fdbd4620baf8acf1574269be" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46037 } }
I20260925 14:58:59.287313 26820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32529958fdbd4620baf8acf1574269be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32529958fdbd4620baf8acf1574269be" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46037 } } }
I20260925 14:58:59.287412 26820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:59.287621 26830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:59.287619 26823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32529958fdbd4620baf8acf1574269be. Latest consensus state: current_term: 1 leader_uuid: "32529958fdbd4620baf8acf1574269be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32529958fdbd4620baf8acf1574269be" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 46037 } } }
I20260925 14:58:59.287840 26823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:59.288790 26830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:59.290782 26830 catalog_manager.cc:1347] Generated new cluster ID: 5fe1e1c775b4485191403352b7843fd5
I20260925 14:58:59.290839 26830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:59.298688 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.286257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.9.141.126:46037)
0925 14:58:59.286428 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.286438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.286746 (+ 308us) server_negotiation.cc:407] Connection header received
0925 14:58:59.286921 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.286955 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.287021 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.287197 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.288270 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.289516 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.290575 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.291182 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.291281 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.294333 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.294337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.294338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.294356 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.296830 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.297283 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.297287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.297287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.297417 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.297648 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.297648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.297649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.297994 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.298137 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.298294 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.298474 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260925 14:58:59.301563 26830 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:59.302079 26830 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:59.314339 26830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be: Generated new TSK 0
I20260925 14:58:59.314555 26830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:58:59.344096 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46037
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.429710 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.429929 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.429955 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.431077 26842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.431118 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.431144 26842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:58:59.433326 26842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:59.433555 26842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.433766 26842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:59.439352 26842 server_base.cc:1034] running on GCE node
W20260925 14:58:59.439364 26850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.439322 26847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.439317 26848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.439781 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.440028 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.441180 26842 hybrid_clock.cc:648] HybridClock initialized: now 1790348339441160 us; error 37 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.443825 26842 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:58:59.444278 26842 webserver.cc:460] Webserver started at http://127.9.141.65:34225/ using document root <none> and password file <none>
I20260925 14:58:59.444577 26842 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.444648 26842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.444789 26842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.445892 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "90a1a799d75b4631808a3ebda37c847a"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "3788739bb3a4fbd92bdc7165c92c4bc0"
server_key_iv: "d3f22be17215a39bac77d20720afd154"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.446270 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "90a1a799d75b4631808a3ebda37c847a"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "3788739bb3a4fbd92bdc7165c92c4bc0"
server_key_iv: "d3f22be17215a39bac77d20720afd154"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.447608 26842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:59.448451 26857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.448652 26842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:59.448792 26842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "90a1a799d75b4631808a3ebda37c847a"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "3788739bb3a4fbd92bdc7165c92c4bc0"
server_key_iv: "d3f22be17215a39bac77d20720afd154"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.448882 26842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.462265 26842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.464828 26842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.464994 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.465250 26842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:59.465626 26842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:59.465682 26842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.465724 26842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:59.465780 26842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.471865 26842 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:34885
I20260925 14:58:59.471907 26970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:34885 every 8 connection(s)
I20260925 14:58:59.472272 26842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:59.479527 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26842
I20260925 14:58:59.479646 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:59.479897 9781 external_mini_cluster.cc:1428] Setting key 1da259b1998ed1f301f65b4fe30661ea
I20260925 14:58:59.482178 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.473070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:50357 (local address 127.9.141.126:46037)
0925 14:58:59.473134 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.473136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.473332 (+ 196us) server_negotiation.cc:407] Connection header received
0925 14:58:59.473893 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.473894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.473908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.473973 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.475079 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.475586 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.476636 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.476828 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.476888 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.479111 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.479113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.479113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.479122 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.480779 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.481184 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.481185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.481186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.481251 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.481517 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.481518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.481519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.481640 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.481718 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.482006 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.482088 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:58:59.482386 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.473033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46037 (local address 127.9.141.65:50357)
0925 14:58:59.473192 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.473243 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.473798 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.474048 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.474052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.474782 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.474986 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.474991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.475670 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.475671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.476563 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.476565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.476905 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.476906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.476999 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.477541 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.477549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.478998 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.480860 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.480867 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.480869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.481095 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.481318 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.481318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.481319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.481442 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.481710 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.481716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.481865 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.481971 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.482138 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260925 14:58:59.483238 26971 heartbeater.cc:347] Connected to a master server at 127.9.141.126:46037
I20260925 14:58:59.483326 26971 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:59.483572 26971 heartbeater.cc:511] Master 127.9.141.126:46037 requested a full tablet report, sending...
I20260925 14:58:59.484179 26780 ts_manager.cc:194] Registered new tserver with Master: 90a1a799d75b4631808a3ebda37c847a (127.9.141.65:34885)
I20260925 14:58:59.484943 26780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:50357
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:58:59.509274 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46037
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.595809 26976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.596025 26976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.596050 26976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.597211 26976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.597251 26976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.597285 26976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:58:59.598709 26976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:59.598906 26976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.599087 26976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.604331 26981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.604449 26984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.604352 26982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.604593 26976 server_base.cc:1034] running on GCE node
I20260925 14:58:59.604844 26976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.605103 26976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.606276 26976 hybrid_clock.cc:648] HybridClock initialized: now 1790348339606242 us; error 45 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.608992 26976 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:58:59.609388 26976 webserver.cc:460] Webserver started at http://127.9.141.66:32881/ using document root <none> and password file <none>
I20260925 14:58:59.609584 26976 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.609625 26976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.609712 26976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.610756 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "6f9b75ea69f1433b81ccd5f5d26564d3"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "ad812ceefc30e9fe5f297c9421f9cc84"
server_key_iv: "488b7861761ad22314fb86a77a86738b"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.611053 26976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "6f9b75ea69f1433b81ccd5f5d26564d3"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "ad812ceefc30e9fe5f297c9421f9cc84"
server_key_iv: "488b7861761ad22314fb86a77a86738b"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.612452 26976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:59.613231 26991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.613399 26976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:59.613487 26976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "6f9b75ea69f1433b81ccd5f5d26564d3"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "ad812ceefc30e9fe5f297c9421f9cc84"
server_key_iv: "488b7861761ad22314fb86a77a86738b"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.613543 26976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.621845 26976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.623991 26976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.624109 26976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.624284 26976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:59.624640 26976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:59.624701 26976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.624745 26976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:59.624800 26976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.630646 26976 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:35795
I20260925 14:58:59.630712 27104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:35795 every 8 connection(s)
I20260925 14:58:59.631057 26976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:59.634735 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 26976
I20260925 14:58:59.634859 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:59.635089 9781 external_mini_cluster.cc:1428] Setting key 87ab06c4d61ac3d4750356be0bd3e6ae
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:59.642431 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.631927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:33057 (local address 127.9.141.126:46037)
0925 14:58:59.631997 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.631999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.632221 (+ 222us) server_negotiation.cc:407] Connection header received
0925 14:58:59.632794 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.632795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.632809 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.632870 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.633980 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.634547 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.636209 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.636499 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.636577 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.639157 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.639159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.639159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.639167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.640824 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.641355 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.641357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.641357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.641424 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.641750 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.641751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.641751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.641881 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.641959 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.642243 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.642332 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:58:59.642643 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.631836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46037 (local address 127.9.141.66:33057)
0925 14:58:59.632038 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.632082 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.632687 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.632981 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.632984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.633679 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.633876 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.633881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.634782 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.634784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.636044 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.636048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.636596 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.636597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.636797 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.637416 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.637429 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.639051 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.640928 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.640936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.640938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.641264 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.641551 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.641552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.641552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.641667 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.641952 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.641959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.642106 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.642223 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.642398 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":109,"threads_started":1}
I20260925 14:58:59.643394 27105 heartbeater.cc:347] Connected to a master server at 127.9.141.126:46037
I20260925 14:58:59.643486 27105 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:59.643692 27105 heartbeater.cc:511] Master 127.9.141.126:46037 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:58:59.644148 26780 ts_manager.cc:194] Registered new tserver with Master: 6f9b75ea69f1433b81ccd5f5d26564d3 (127.9.141.66:35795)
I20260925 14:58:59.644658 26780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:33057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:58:59.664572 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46037
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.747685 27110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.747905 27110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.747931 27110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.749246 27110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.749297 27110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.749326 27110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:58:59.750886 27110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:58:59.751140 27110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.751426 27110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.756863 27115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.756887 27116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.756877 27118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.756933 27110 server_base.cc:1034] running on GCE node
I20260925 14:58:59.757436 27110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.757740 27110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.758912 27110 hybrid_clock.cc:648] HybridClock initialized: now 1790348339758890 us; error 43 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.761812 27110 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:58:59.762395 27110 webserver.cc:460] Webserver started at http://127.9.141.67:34801/ using document root <none> and password file <none>
I20260925 14:58:59.762709 27110 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.762794 27110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.762975 27110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.764146 27110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "c97a2960bab24afb9efa2b7f751c203a"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "0e90bafc746e375477d7d6a277a9ab88"
server_key_iv: "a238baee1568e173e0a6839c84d17c61"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.764568 27110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "c97a2960bab24afb9efa2b7f751c203a"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "0e90bafc746e375477d7d6a277a9ab88"
server_key_iv: "a238baee1568e173e0a6839c84d17c61"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.766155 27110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:59.767055 27125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.767258 27110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:59.767372 27110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "c97a2960bab24afb9efa2b7f751c203a"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-1sv7"
server_key: "0e90bafc746e375477d7d6a277a9ab88"
server_key_iv: "a238baee1568e173e0a6839c84d17c61"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.767448 27110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.777086 27110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.779120 27110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.779286 27110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.779515 27110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:59.779830 27110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:59.779886 27110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.779929 27110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:59.779958 27110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.785940 27110 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:36347
I20260925 14:58:59.785987 27238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:36347 every 8 connection(s)
I20260925 14:58:59.786331 27110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:59.790899 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27110
I20260925 14:58:59.791020 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:59.791219 9781 external_mini_cluster.cc:1428] Setting key 24ba90d65e441d7e5dfdfc885d8381a2
Sep 25 14:58:59 dist-test-slave-1sv7 krb5kdc[26735](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:58:59.796890 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.787173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:44805 (local address 127.9.141.126:46037)
0925 14:58:59.787442 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.787443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.787530 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:58:59.788133 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.788134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.788144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.788193 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.789405 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.789886 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.790974 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.791287 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.791365 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.793957 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.793958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.793959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.793965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.795605 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.795994 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.795995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.795995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.796049 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.796270 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.796271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.796271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.796433 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.796496 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.796757 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.796822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260925 14:58:59.797127 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.787174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:46037 (local address 127.9.141.67:44805)
0925 14:58:59.787370 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.787425 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.788033 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.788257 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.788260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.788969 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.789301 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.789308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.789969 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.789970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.790890 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.790892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.791376 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.791377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.791550 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.792227 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.792235 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.793858 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.795693 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.795698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.795700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.795924 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.796113 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.796113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.796114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.796209 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.796496 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.796503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.796629 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.796723 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.796862 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260925 14:58:59.797704 27239 heartbeater.cc:347] Connected to a master server at 127.9.141.126:46037
I20260925 14:58:59.797811 27239 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:59.797969 27239 heartbeater.cc:511] Master 127.9.141.126:46037 requested a full tablet report, sending...
I20260925 14:58:59.798377 26780 ts_manager.cc:194] Registered new tserver with Master: c97a2960bab24afb9efa2b7f751c203a (127.9.141.67:36347)
I20260925 14:58:59.798758 26780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:44805
I20260925 14:58:59.803115 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:59.810539 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.804530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.9.141.126:46037)
0925 14:58:59.804593 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.804599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.804638 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:58:59.804700 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.804700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.804709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.804766 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.805261 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.805765 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.806546 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.806753 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.806811 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.808041 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.808042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.808043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.808049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.809377 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.809738 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.809739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.809739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.809795 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.810033 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.810034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.810034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.810195 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.810319 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.810415 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.810474 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:59.812536 26780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:59.814196 26780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:59.820981 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.816941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.9.141.67:36347)
0925 14:58:59.817330 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.817335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.817347 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:58:59.817413 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.817416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.817434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.817594 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.817789 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.818486 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.819850 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.820567 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.820663 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.820725 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.820853 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.820892 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260925 14:58:59.820972 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.816751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:36347 (local address 127.0.0.1:45060)
0925 14:58:59.816967 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.817011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.817087 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.817551 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.817554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.817557 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.817728 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.817730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.818589 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.818590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.819760 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.819763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.820665 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.820666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.820744 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.820758 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.820841 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.820895 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":125,"thread_start_us":53,"threads_started":1}
I20260925 14:58:59.821198 27250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.816943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:34885 (local address 127.0.0.1:49286)
0925 14:58:59.817193 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.817203 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.817241 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.817556 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.817557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.817559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.817652 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.817654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.819077 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.819078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.820071 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.820073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.820981 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.820982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.821044 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.821049 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.821080 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.821119 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260925 14:58:59.821524 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.816750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:35795 (local address 127.0.0.1:39198)
0925 14:58:59.817146 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.817156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.817207 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.817645 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.817646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.817648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.817694 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.817696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.819027 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.819028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.820082 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.820084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.821284 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.821285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.821349 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.821353 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.821425 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.821462 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260925 14:58:59.821576 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.816873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.9.141.66:35795)
0925 14:58:59.817427 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.817431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.817440 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:59.817476 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.817478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.817492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.817565 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.817763 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.818911 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.820451 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.821201 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.821277 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.821340 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.821451 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.821505 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":92,"threads_started":1}
I20260925 14:58:59.821762 27173 tablet_service.cc:1467] Processing CreateTablet for tablet fb453ba80aae43ff89ac3f3c374de45d (DEFAULT_TABLE table=test-table [id=342075fd72814116a5c01b1b3c742873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:59.822167 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.817152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.9.141.65:34885)
0925 14:58:59.817342 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.817345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.817353 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:59.817415 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.817417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.817432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.817825 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.817897 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.818950 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.820163 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.820878 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.821952 (+ 1074us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.822041 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.822066 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.822110 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260925 14:58:59.822178 27173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb453ba80aae43ff89ac3f3c374de45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.822216 27039 tablet_service.cc:1467] Processing CreateTablet for tablet fb453ba80aae43ff89ac3f3c374de45d (DEFAULT_TABLE table=test-table [id=342075fd72814116a5c01b1b3c742873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:59.822542 27039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb453ba80aae43ff89ac3f3c374de45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.822897 26905 tablet_service.cc:1467] Processing CreateTablet for tablet fb453ba80aae43ff89ac3f3c374de45d (DEFAULT_TABLE table=test-table [id=342075fd72814116a5c01b1b3c742873]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:59.823223 26905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb453ba80aae43ff89ac3f3c374de45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.825706 27256 tablet_bootstrap.cc:492] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Bootstrap starting.
I20260925 14:58:59.826074 27257 tablet_bootstrap.cc:492] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Bootstrap starting.
I20260925 14:58:59.826404 27258 tablet_bootstrap.cc:492] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: Bootstrap starting.
I20260925 14:58:59.826637 27256 tablet_bootstrap.cc:654] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.826731 27257 tablet_bootstrap.cc:654] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.827060 27258 tablet_bootstrap.cc:654] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.827221 27256 log.cc:824] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.827375 27257 log.cc:824] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.827631 27258 log.cc:824] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.828094 27256 tablet_bootstrap.cc:492] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: No bootstrap required, opened a new log
I20260925 14:58:59.828157 27256 ts_tablet_manager.cc:1397] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:59.828352 27258 tablet_bootstrap.cc:492] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: No bootstrap required, opened a new log
I20260925 14:58:59.828455 27258 ts_tablet_manager.cc:1397] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:59.828905 27257 tablet_bootstrap.cc:492] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: No bootstrap required, opened a new log
I20260925 14:58:59.828970 27257 ts_tablet_manager.cc:1397] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:59.829651 27256 raft_consensus.cc:348] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.829790 27256 raft_consensus.cc:374] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.829813 27256 raft_consensus.cc:729] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97a2960bab24afb9efa2b7f751c203a, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.829941 27256 consensus_queue.cc:260] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.830238 27239 heartbeater.cc:503] Master 127.9.141.126:46037 was elected leader, sending a full tablet report...
I20260925 14:58:59.830251 27256 ts_tablet_manager.cc:1428] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:59.830444 27258 raft_consensus.cc:348] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.830603 27258 raft_consensus.cc:374] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.830641 27258 raft_consensus.cc:729] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a1a799d75b4631808a3ebda37c847a, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.830744 27258 consensus_queue.cc:260] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.831013 27258 ts_tablet_manager.cc:1428] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:59.830901 27257 raft_consensus.cc:348] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.831064 27257 raft_consensus.cc:374] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.831097 27257 raft_consensus.cc:729] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9b75ea69f1433b81ccd5f5d26564d3, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.831148 26971 heartbeater.cc:503] Master 127.9.141.126:46037 was elected leader, sending a full tablet report...
I20260925 14:58:59.831198 27257 consensus_queue.cc:260] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.831472 27257 ts_tablet_manager.cc:1428] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:59.831637 27105 heartbeater.cc:503] Master 127.9.141.126:46037 was elected leader, sending a full tablet report...
W20260925 14:58:59.881726 27106 tablet.cc:2369] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:59.932581 27262 raft_consensus.cc:482] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:59.932717 27262 raft_consensus.cc:504] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.933153 27262 leader_election.cc:288] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f9b75ea69f1433b81ccd5f5d26564d3 (127.9.141.66:35795), 90a1a799d75b4631808a3ebda37c847a (127.9.141.65:34885)
I20260925 14:58:59.936573 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.933522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:45449 (local address 127.9.141.65:34885)
0925 14:58:59.933595 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.933597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.933607 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:59.933639 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.933641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.933651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.933756 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.933951 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.934674 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.935642 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.936166 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.936248 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.936258 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.936447 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.936487 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:58:59.936568 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.933367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:34885 (local address 127.9.141.67:45449)
0925 14:58:59.933445 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.933458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.933526 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.933789 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.933790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.933792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.933871 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.933873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.934775 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.934775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.935542 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.935544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.936259 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.936260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.936340 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.936344 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.936435 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.936481 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:58:59.936847 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.933530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:48271 (local address 127.9.141.66:35795)
0925 14:58:59.933594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.933597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.933898 (+ 301us) server_negotiation.cc:407] Connection header received
0925 14:58:59.933977 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.933978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.933993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.934038 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.934235 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.934882 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.935877 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.936480 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.936548 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.936555 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.936744 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.936786 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:58:59.936981 26925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb453ba80aae43ff89ac3f3c374de45d" candidate_uuid: "c97a2960bab24afb9efa2b7f751c203a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a1a799d75b4631808a3ebda37c847a" is_pre_election: true
I20260925 14:58:59.937104 27265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.933367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:35795 (local address 127.9.141.67:48271)
0925 14:58:59.933817 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.933826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.933889 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.934072 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.934073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.934074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.934139 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.934141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.934968 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.934970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.935747 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.935750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.936560 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.936561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.936645 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.936649 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.936950 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.937000 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":242,"thread_start_us":113,"threads_started":1}
I20260925 14:58:59.937158 26925 raft_consensus.cc:2454] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97a2960bab24afb9efa2b7f751c203a in term 0.
I20260925 14:58:59.937470 27129 leader_election.cc:302] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90a1a799d75b4631808a3ebda37c847a, c97a2960bab24afb9efa2b7f751c203a; no voters:
I20260925 14:58:59.937439 27059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb453ba80aae43ff89ac3f3c374de45d" candidate_uuid: "c97a2960bab24afb9efa2b7f751c203a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" is_pre_election: true
I20260925 14:58:59.937587 27059 raft_consensus.cc:2454] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97a2960bab24afb9efa2b7f751c203a in term 0.
I20260925 14:58:59.937603 27262 raft_consensus.cc:2781] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:59.937655 27262 raft_consensus.cc:482] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:59.937688 27262 raft_consensus.cc:3037] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.938398 27262 raft_consensus.cc:504] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.938542 27262 leader_election.cc:288] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [CANDIDATE]: Term 1 election: Requested vote from peers 6f9b75ea69f1433b81ccd5f5d26564d3 (127.9.141.66:35795), 90a1a799d75b4631808a3ebda37c847a (127.9.141.65:34885)
I20260925 14:58:59.938704 27059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb453ba80aae43ff89ac3f3c374de45d" candidate_uuid: "c97a2960bab24afb9efa2b7f751c203a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3"
I20260925 14:58:59.938707 26925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb453ba80aae43ff89ac3f3c374de45d" candidate_uuid: "c97a2960bab24afb9efa2b7f751c203a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90a1a799d75b4631808a3ebda37c847a"
I20260925 14:58:59.938784 27059 raft_consensus.cc:3037] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.938804 26925 raft_consensus.cc:3037] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.939476 27059 raft_consensus.cc:2454] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97a2960bab24afb9efa2b7f751c203a in term 1.
I20260925 14:58:59.939634 26925 raft_consensus.cc:2454] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97a2960bab24afb9efa2b7f751c203a in term 1.
I20260925 14:58:59.939646 27126 leader_election.cc:302] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f9b75ea69f1433b81ccd5f5d26564d3, c97a2960bab24afb9efa2b7f751c203a; no voters:
I20260925 14:58:59.939759 27262 raft_consensus.cc:2781] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:59.939910 27262 raft_consensus.cc:686] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 1 LEADER]: Becoming Leader. State: Replica: c97a2960bab24afb9efa2b7f751c203a, State: Running, Role: LEADER
I20260925 14:58:59.940011 27262 consensus_queue.cc:237] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } }
I20260925 14:58:59.940913 26779 catalog_manager.cc:5495] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a reported cstate change: term changed from 0 to 1, leader changed from <none> to c97a2960bab24afb9efa2b7f751c203a (127.9.141.67). New cstate: current_term: 1 leader_uuid: "c97a2960bab24afb9efa2b7f751c203a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c97a2960bab24afb9efa2b7f751c203a" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:59.969981 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.967468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.9.141.67:36347)
0925 14:58:59.967520 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.967522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.967529 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:59.967580 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.967581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.967598 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.967680 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:59.967811 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.968284 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.969107 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.969334 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.969424 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.969595 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:59.969712 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:59.969811 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.969830 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.969851 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:58:59.971915 27059 raft_consensus.cc:1261] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 1 FOLLOWER]: Refusing update from remote peer c97a2960bab24afb9efa2b7f751c203a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:59.971922 26925 raft_consensus.cc:1261] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 1 FOLLOWER]: Refusing update from remote peer c97a2960bab24afb9efa2b7f751c203a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:59.972301 27266 consensus_queue.cc:1035] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [LEADER]: Connected to new peer: Peer: permanent_uuid: "90a1a799d75b4631808a3ebda37c847a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 34885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:59.972482 27262 consensus_queue.cc:1035] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f9b75ea69f1433b81ccd5f5d26564d3" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 35795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:59.973827 26972 tablet.cc:2369] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:59.975981 27270 mvcc.cc:205] Tried to move back new op lower bound from 7333266800522289152 to 7333266800395071488. Current Snapshot: MvccSnapshot[applied={T|T < 7333266800522289152 or (T in {7333266800522289152})}]
I20260925 14:58:59.976104 27271 mvcc.cc:205] Tried to move back new op lower bound from 7333266800522289152 to 7333266800395071488. Current Snapshot: MvccSnapshot[applied={T|T < 7333266800522289152 or (T in {7333266800522289152})}]
I20260925 14:58:59.977259 26779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:59.977365 26779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:59.978163 26779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32529958fdbd4620baf8acf1574269be: Sending DeleteTablet for 3 replicas of tablet fb453ba80aae43ff89ac3f3c374de45d
I20260925 14:58:59.978523 27039 tablet_service.cc:1514] Processing DeleteTablet for tablet fb453ba80aae43ff89ac3f3c374de45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:59 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:39198
I20260925 14:58:59.978602 26905 tablet_service.cc:1514] Processing DeleteTablet for tablet fb453ba80aae43ff89ac3f3c374de45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:59 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:49286
I20260925 14:58:59.978600 27173 tablet_service.cc:1514] Processing DeleteTablet for tablet fb453ba80aae43ff89ac3f3c374de45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:59 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:45060
I20260925 14:58:59.978826 27279 tablet_replica.cc:330] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: stopping tablet replica
I20260925 14:58:59.978930 27279 raft_consensus.cc:2229] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:59.979031 27279 raft_consensus.cc:2258] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:59.979067 27278 tablet_replica.cc:330] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: stopping tablet replica
I20260925 14:58:59.979154 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26842
I20260925 14:58:59.979166 27278 raft_consensus.cc:2229] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:59.979177 27280 tablet_replica.cc:330] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a: stopping tablet replica
I20260925 14:58:59.979265 27280 raft_consensus.cc:2229] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:59.979264 27278 raft_consensus.cc:2258] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:59.979354 27280 raft_consensus.cc:2258] T fb453ba80aae43ff89ac3f3c374de45d P 90a1a799d75b4631808a3ebda37c847a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:59.979969 27279 ts_tablet_manager.cc:1905] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:59.980669 27278 ts_tablet_manager.cc:1905] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:59.981951 27279 ts_tablet_manager.cc:1918] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:59.982018 27279 log.cc:1196] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/fb453ba80aae43ff89ac3f3c374de45d
I20260925 14:58:59.982293 27279 ts_tablet_manager.cc:1939] T fb453ba80aae43ff89ac3f3c374de45d P c97a2960bab24afb9efa2b7f751c203a: Deleting consensus metadata
I20260925 14:58:59.982589 27278 ts_tablet_manager.cc:1918] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:59.982653 27278 log.cc:1196] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/fb453ba80aae43ff89ac3f3c374de45d
I20260925 14:58:59.982841 26767 catalog_manager.cc:4841] TS c97a2960bab24afb9efa2b7f751c203a (127.9.141.67:36347): tablet fb453ba80aae43ff89ac3f3c374de45d (table test-table [id=342075fd72814116a5c01b1b3c742873]) successfully deleted
I20260925 14:58:59.982923 27278 ts_tablet_manager.cc:1939] T fb453ba80aae43ff89ac3f3c374de45d P 6f9b75ea69f1433b81ccd5f5d26564d3: Deleting consensus metadata
I20260925 14:58:59.983842 26764 catalog_manager.cc:4841] TS 6f9b75ea69f1433b81ccd5f5d26564d3 (127.9.141.66:35795): tablet fb453ba80aae43ff89ac3f3c374de45d (table test-table [id=342075fd72814116a5c01b1b3c742873]) successfully deleted
W20260925 14:58:59.986899 26767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:34885 (error 108)
I20260925 14:58:59.987125 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26976
I20260925 14:58:59.987444 27253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.987317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:34885 (local address 127.0.0.1:49288)
0925 14:58:59.987369 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.987402 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:34885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:58:59.988408 26767 catalog_manager.cc:4584] TS 90a1a799d75b4631808a3ebda37c847a (127.9.141.65:34885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb453ba80aae43ff89ac3f3c374de45d: Network error: Client connection negotiation failed: client connection to 127.9.141.65:34885: connect: Connection refused (error 111)
I20260925 14:58:59.992759 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27110
I20260925 14:58:59.997251 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 26748
2026-09-25T14:59:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2060 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T14:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:00Z Disabled control of system clock
I20260925 14:59:00.025614 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:40647
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:00.106907 27285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:00.107129 27285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:00.107149 27285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:00.108314 27285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:00.108357 27285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:00.108371 27285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:00.108453 27285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:00.108469 27285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:00.109915 27285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:40647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:00.110176 27285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:00.110469 27285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:00.115358 27293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.115386 27291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.115391 27290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.115552 27285 server_base.cc:1034] running on GCE node
I20260925 14:59:00.115842 27285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:00.116137 27285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:00.117311 27285 hybrid_clock.cc:648] HybridClock initialized: now 1790348340117305 us; error 40 us; skew 500 ppm
I20260925 14:59:00.118637 27285 webserver.cc:460] Webserver started at http://127.9.141.126:34547/ using document root <none> and password file <none>
I20260925 14:59:00.118892 27285 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:00.118963 27285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:00.119124 27285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:00.120306 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "b9ed43091c5e1bc1fe2301d07dc23fb4"
server_key_iv: "d8efa0ee4b957f38401f4cea2809fddf"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.120739 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "b9ed43091c5e1bc1fe2301d07dc23fb4"
server_key_iv: "d8efa0ee4b957f38401f4cea2809fddf"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.122030 27285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:00.122812 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.123008 27285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:00.123113 27285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "b9ed43091c5e1bc1fe2301d07dc23fb4"
server_key_iv: "d8efa0ee4b957f38401f4cea2809fddf"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.123191 27285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.131666 27285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:00.131971 27285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:00.132136 27285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:00.136145 27285 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:40647
I20260925 14:59:00.136174 27351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:40647 every 8 connection(s)
I20260925 14:59:00.136615 27285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:00.137588 27352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:00.139853 27352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90: Bootstrap starting.
I20260925 14:59:00.140218 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27285
I20260925 14:59:00.140332 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:00.140576 9781 external_mini_cluster.cc:1428] Setting key 93c76923367431ebd4092bfa57e8159e
I20260925 14:59:00.140618 27352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:00.140920 27352 log.cc:824] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:00.141753 27352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90: No bootstrap required, opened a new log
I20260925 14:59:00.143682 27352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40647 } }
I20260925 14:59:00.143894 27352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:00.143932 27352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8643695f3c2a4d3b8acf6dd2f21b1b90, State: Initialized, Role: FOLLOWER
I20260925 14:59:00.144088 27352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40647 } }
I20260925 14:59:00.144160 27352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:00.144186 27352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:00.144223 27352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:00.144860 27352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40647 } }
I20260925 14:59:00.144948 27352 leader_election.cc:302] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8643695f3c2a4d3b8acf6dd2f21b1b90; no voters:
I20260925 14:59:00.145147 27352 leader_election.cc:288] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:00.145205 27357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:00.145354 27357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [term 1 LEADER]: Becoming Leader. State: Replica: 8643695f3c2a4d3b8acf6dd2f21b1b90, State: Running, Role: LEADER
I20260925 14:59:00.145437 27352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:00.145452 27357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40647 } }
I20260925 14:59:00.145802 27358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40647 } } }
I20260925 14:59:00.145891 27358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:00.146104 27363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:00.146144 27359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8643695f3c2a4d3b8acf6dd2f21b1b90. Latest consensus state: current_term: 1 leader_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8643695f3c2a4d3b8acf6dd2f21b1b90" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 40647 } } }
I20260925 14:59:00.146317 27359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:00.146605 27363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:00.148134 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.141142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.9.141.126:40647)
0925 14:59:00.141341 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.141349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.141394 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:59:00.141969 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.141978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.142061 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.142229 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.142385 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.144414 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.145389 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.146030 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.146200 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.147162 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.147167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.147167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:00.147401 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.147450 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:00.147467 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.147531 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.147797 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.147894 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260925 14:59:00.148633 27363 catalog_manager.cc:1347] Generated new cluster ID: 75b1d145be20408ea37bb0a03b04f12a
I20260925 14:59:00.148682 27363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:00.162313 27363 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:00.163074 27363 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:00.167357 27363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90: Generated new TSK 0
I20260925 14:59:00.167599 27363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:00.174768 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40647
--builtin_ntp_servers=127.9.141.84:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:00.262869 27376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:00.263118 27376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:00.263154 27376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:00.264351 27376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:00.264447 27376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:00.264492 27376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:59:00.265892 27376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:00.266172 27376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:00.266496 27376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:00.271551 27381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.271687 27382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.271586 27384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.272466 27376 server_base.cc:1034] running on GCE node
I20260925 14:59:00.272691 27376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:00.272958 27376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:00.274128 27376 hybrid_clock.cc:648] HybridClock initialized: now 1790348340274104 us; error 40 us; skew 500 ppm
I20260925 14:59:00.278712 27376 webserver.cc:460] Webserver started at http://127.9.141.65:38217/ using document root <none> and password file <none>
I20260925 14:59:00.278997 27376 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:00.279074 27376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:00.279390 27376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:00.280737 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "098c3ec1270a47cf81d8bc71997a8e43"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "1ab58f7f7759d5268069ec44f4e455f4"
server_key_iv: "dfea0f206a530f10165d872108454206"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.281103 27376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "098c3ec1270a47cf81d8bc71997a8e43"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "1ab58f7f7759d5268069ec44f4e455f4"
server_key_iv: "dfea0f206a530f10165d872108454206"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.282465 27376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:00.283353 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.283648 27376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:00.283766 27376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "098c3ec1270a47cf81d8bc71997a8e43"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "1ab58f7f7759d5268069ec44f4e455f4"
server_key_iv: "dfea0f206a530f10165d872108454206"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.283855 27376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.299554 27376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:00.299901 27376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:00.300060 27376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:00.300361 27376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:00.300818 27376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:00.300880 27376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.300926 27376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:00.300951 27376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.306716 27376 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:38445
I20260925 14:59:00.306761 27503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:38445 every 8 connection(s)
I20260925 14:59:00.307149 27376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:00.310122 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27376
I20260925 14:59:00.310235 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:00.310495 9781 external_mini_cluster.cc:1428] Setting key 309fa5555d73ff0caa43c66edece7fde
I20260925 14:59:00.311466 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40647
--builtin_ntp_servers=127.9.141.84:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:59:00.313544 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.307980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:53777 (local address 127.9.141.126:40647)
0925 14:59:00.308025 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.308027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.308318 (+ 291us) server_negotiation.cc:407] Connection header received
0925 14:59:00.308894 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.308896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.308906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.308972 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.309729 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.310310 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.311897 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.312083 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.312153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.313212 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.313214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.313215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:00.313220 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.313239 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:00.313244 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.313320 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.313413 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.313442 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:00.313684 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.307941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40647 (local address 127.9.141.65:53777)
0925 14:59:00.308130 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.308172 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.308793 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.309056 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.309059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.309068 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.309632 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.309637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.310420 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.310421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.311803 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.311807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.312183 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.312184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.312361 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.313021 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:00.313022 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.313065 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:00.313098 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.313331 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.313336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.313339 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.313395 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.313480 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260925 14:59:00.314218 27504 heartbeater.cc:347] Connected to a master server at 127.9.141.126:40647
I20260925 14:59:00.314327 27504 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:00.314527 27504 heartbeater.cc:511] Master 127.9.141.126:40647 requested a full tablet report, sending...
I20260925 14:59:00.315039 27316 ts_manager.cc:194] Registered new tserver with Master: 098c3ec1270a47cf81d8bc71997a8e43 (127.9.141.65:38445)
I20260925 14:59:00.315663 27316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.65:53777
W20260925 14:59:00.394260 27507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:00.394526 27507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:00.394564 27507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:00.395771 27507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:00.395851 27507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:00.395900 27507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:59:00.397399 27507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:00.397680 27507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:00.398020 27507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:00.403157 27513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.403146 27515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.403143 27512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.403439 27507 server_base.cc:1034] running on GCE node
I20260925 14:59:00.403720 27507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:00.404008 27507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:00.405212 27507 hybrid_clock.cc:648] HybridClock initialized: now 1790348340405190 us; error 39 us; skew 500 ppm
I20260925 14:59:00.406668 27507 webserver.cc:460] Webserver started at http://127.9.141.66:42917/ using document root <none> and password file <none>
I20260925 14:59:00.406942 27507 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:00.406993 27507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:00.407123 27507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:00.408214 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "25811f96fa2447458af3f38cc171344f"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "c36713f23202711b3c47f89b16aecb96"
server_key_iv: "5ae84cdaad87e2d5d633ced3cdd28761"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.408659 27507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "25811f96fa2447458af3f38cc171344f"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "c36713f23202711b3c47f89b16aecb96"
server_key_iv: "5ae84cdaad87e2d5d633ced3cdd28761"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.409996 27507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:00.410756 27521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.410972 27507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:59:00.411063 27507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "25811f96fa2447458af3f38cc171344f"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "c36713f23202711b3c47f89b16aecb96"
server_key_iv: "5ae84cdaad87e2d5d633ced3cdd28761"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.411118 27507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.427959 27507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:00.428236 27507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:00.428340 27507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:00.428601 27507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:00.428893 27507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:00.428925 27507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.428946 27507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:00.428959 27507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.434419 27507 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:34851
I20260925 14:59:00.434506 27634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:34851 every 8 connection(s)
I20260925 14:59:00.434753 27507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:00.437489 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27507
I20260925 14:59:00.437579 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:00.437738 9781 external_mini_cluster.cc:1428] Setting key e94d39d818285b31166dd2b13c84e1bc
I20260925 14:59:00.438711 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40647
--builtin_ntp_servers=127.9.141.84:40705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:59:00.440641 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.435567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:48571 (local address 127.9.141.126:40647)
0925 14:59:00.435612 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.435615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.435846 (+ 231us) server_negotiation.cc:407] Connection header received
0925 14:59:00.436329 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.436330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.436339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.436473 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.437171 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.437848 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.439135 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.439353 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.439434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.440357 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.440358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.440359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:00.440364 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.440403 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:00.440408 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.440461 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.440537 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.440563 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:00.440899 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.435529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40647 (local address 127.9.141.66:48571)
0925 14:59:00.435709 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.435752 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.436229 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.436542 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.436545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.436553 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.437082 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.437086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.437952 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.437954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.439045 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.439048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.439441 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.439442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.439592 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.440197 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:00.440202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.440232 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:00.440257 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.440465 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.440470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.440473 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.440542 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.440643 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260925 14:59:00.441344 27635 heartbeater.cc:347] Connected to a master server at 127.9.141.126:40647
I20260925 14:59:00.441459 27635 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:00.441648 27635 heartbeater.cc:511] Master 127.9.141.126:40647 requested a full tablet report, sending...
I20260925 14:59:00.442109 27316 ts_manager.cc:194] Registered new tserver with Master: 25811f96fa2447458af3f38cc171344f (127.9.141.66:34851)
I20260925 14:59:00.442639 27316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.66:48571
W20260925 14:59:00.520768 27638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:00.520977 27638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:00.520999 27638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:00.522114 27638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:00.522146 27638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:00.522171 27638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:59:00.523465 27638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:00.523671 27638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:00.523932 27638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:00.529387 27643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.529381 27646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:00.529376 27644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:00.529748 27638 server_base.cc:1034] running on GCE node
I20260925 14:59:00.529956 27638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:00.530258 27638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:00.531419 27638 hybrid_clock.cc:648] HybridClock initialized: now 1790348340531409 us; error 35 us; skew 500 ppm
I20260925 14:59:00.532876 27638 webserver.cc:460] Webserver started at http://127.9.141.67:40893/ using document root <none> and password file <none>
I20260925 14:59:00.533159 27638 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:00.533300 27638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:00.533474 27638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:00.534575 27638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "33c0506cfa4345b49b8f3b1c4cf255f0"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "ec4a5d52e4586f51c24e6d83180ffce4"
server_key_iv: "1e5d448420b19aedf63ef321e07227e5"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.534941 27638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "33c0506cfa4345b49b8f3b1c4cf255f0"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "ec4a5d52e4586f51c24e6d83180ffce4"
server_key_iv: "1e5d448420b19aedf63ef321e07227e5"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.536633 27638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:00.537663 27652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.537919 27638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:00.538046 27638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "33c0506cfa4345b49b8f3b1c4cf255f0"
format_stamp: "Formatted at 2026-09-25 14:59:00 on dist-test-slave-1sv7"
server_key: "ec4a5d52e4586f51c24e6d83180ffce4"
server_key_iv: "1e5d448420b19aedf63ef321e07227e5"
server_key_version: "encryptionkey@0"
I20260925 14:59:00.538133 27638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:00.553586 27638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:00.553943 27638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:00.554111 27638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:00.554364 27638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:00.554695 27638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:00.554750 27638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.554791 27638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:00.554821 27638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:00.560937 27638 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:33187
I20260925 14:59:00.560983 27765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:33187 every 8 connection(s)
I20260925 14:59:00.561326 27638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:00.564061 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27638
I20260925 14:59:00.564179 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:00.564430 9781 external_mini_cluster.cc:1428] Setting key c6607778ce72457be86447a93225d6ce
I20260925 14:59:00.567250 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.562106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:56843 (local address 127.9.141.126:40647)
0925 14:59:00.562167 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.562169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.562478 (+ 309us) server_negotiation.cc:407] Connection header received
0925 14:59:00.563038 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.563040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.563052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.563111 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.563823 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.564437 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.565779 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.565987 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.566073 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.566989 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.566991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.566991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:00.566997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.567013 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:00.567017 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.567062 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.567141 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.567168 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:59:00.567494 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.562077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:40647 (local address 127.9.141.67:56843)
0925 14:59:00.562350 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.562385 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.562928 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.563187 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.563191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.563199 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:00.563723 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.563728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.564532 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.564533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.565686 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.565689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.566083 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.566084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.566234 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.566842 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:00.566843 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:00.566872 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:00.566896 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:00.567073 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:00.567078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:00.567081 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.567146 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.567258 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260925 14:59:00.567963 27766 heartbeater.cc:347] Connected to a master server at 127.9.141.126:40647
I20260925 14:59:00.568056 27766 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:00.568265 27766 heartbeater.cc:511] Master 127.9.141.126:40647 requested a full tablet report, sending...
I20260925 14:59:00.568693 27316 ts_manager.cc:194] Registered new tserver with Master: 33c0506cfa4345b49b8f3b1c4cf255f0 (127.9.141.67:33187)
I20260925 14:59:00.569082 27316 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.67:56843
I20260925 14:59:00.576717 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:00.581269 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.578125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.9.141.126:40647)
0925 14:59:00.578221 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.578224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.578318 (+ 94us) server_negotiation.cc:407] Connection header received
0925 14:59:00.578449 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.578450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.578461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.578544 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:00.578712 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.579219 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.580008 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.580205 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.580293 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.581006 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:00.581008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:00.581008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:00.581013 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:00.581034 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:00.581038 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:00.581100 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.581156 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.581179 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:59:00.583003 27316 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:00.584834 27316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:00.591470 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.587000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:38445 (local address 127.0.0.1:34254)
0925 14:59:00.587213 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.587260 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.587337 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.588026 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.588027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.588029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:00.588079 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.588080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.589015 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.589015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.590505 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.590506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.591169 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.591170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.591244 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.591257 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.591338 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.591394 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":7,"thread_start_us":86,"threads_started":1}
I20260925 14:59:00.591480 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.587087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.9.141.65:38445)
0925 14:59:00.587876 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.587880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.587890 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:00.587932 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.587934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.587951 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.588029 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:00.588135 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.588922 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.590585 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.591079 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.591169 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.591265 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.591354 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.591392 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":78,"threads_started":1}
I20260925 14:59:00.591640 27774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.587039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:33187 (local address 127.0.0.1:45660)
0925 14:59:00.587258 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.587270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.587378 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.587643 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.587646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.587649 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:00.587733 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.587735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.589230 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.589231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.590317 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.590319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.591363 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.591364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.591437 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.591441 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.591498 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.591560 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260925 14:59:00.591665 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.587447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.9.141.66:34851)
0925 14:59:00.587610 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.587614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.587741 (+ 127us) server_negotiation.cc:407] Connection header received
0925 14:59:00.587818 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.587820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.587836 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.587916 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:00.588111 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.589343 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.590349 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.591145 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.591232 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.591320 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.591543 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.591580 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260925 14:59:00.591868 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.587251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34851 (local address 127.0.0.1:38548)
0925 14:59:00.587669 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.587680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.587737 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.587983 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.587984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.587986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:00.588027 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.588029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.589432 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.589433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.590237 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.590239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.591364 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.591365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.591439 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.591442 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.591498 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.591792 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":243,"thread_start_us":69,"threads_started":1}
I20260925 14:59:00.592126 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.587236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.9.141.67:33187)
0925 14:59:00.587456 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.587461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.587471 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:00.587518 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.587519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.587533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.587625 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:00.587828 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.589138 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.590386 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.591145 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.591227 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.591320 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.591960 (+ 640us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.591994 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
I20260925 14:59:00.592729 27438 tablet_service.cc:1467] Processing CreateTablet for tablet abcb6536506e40a9bb6fa1736ba3b7c9 (DEFAULT_TABLE table=test-table [id=d09c8b90cdfe4d3cbea42f39586cc75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:00.592706 27569 tablet_service.cc:1467] Processing CreateTablet for tablet abcb6536506e40a9bb6fa1736ba3b7c9 (DEFAULT_TABLE table=test-table [id=d09c8b90cdfe4d3cbea42f39586cc75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:00.592839 27700 tablet_service.cc:1467] Processing CreateTablet for tablet abcb6536506e40a9bb6fa1736ba3b7c9 (DEFAULT_TABLE table=test-table [id=d09c8b90cdfe4d3cbea42f39586cc75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:00.593040 27438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcb6536506e40a9bb6fa1736ba3b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:00.593060 27569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcb6536506e40a9bb6fa1736ba3b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:00.593109 27700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcb6536506e40a9bb6fa1736ba3b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:00.595865 27780 tablet_bootstrap.cc:492] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Bootstrap starting.
I20260925 14:59:00.596467 27782 tablet_bootstrap.cc:492] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Bootstrap starting.
I20260925 14:59:00.596830 27780 tablet_bootstrap.cc:654] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:00.597211 27782 tablet_bootstrap.cc:654] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:00.597230 27780 log.cc:824] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:00.597565 27782 log.cc:824] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:00.597648 27781 tablet_bootstrap.cc:492] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Bootstrap starting.
I20260925 14:59:00.598038 27780 tablet_bootstrap.cc:492] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: No bootstrap required, opened a new log
I20260925 14:59:00.598134 27780 ts_tablet_manager.cc:1397] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:00.598315 27781 tablet_bootstrap.cc:654] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:00.598367 27782 tablet_bootstrap.cc:492] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: No bootstrap required, opened a new log
I20260925 14:59:00.598443 27782 ts_tablet_manager.cc:1397] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:59:00.598667 27781 log.cc:824] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:00.599551 27781 tablet_bootstrap.cc:492] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: No bootstrap required, opened a new log
I20260925 14:59:00.599624 27781 ts_tablet_manager.cc:1397] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:00.600728 27780 raft_consensus.cc:348] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.600884 27780 raft_consensus.cc:374] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:00.600927 27780 raft_consensus.cc:729] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 098c3ec1270a47cf81d8bc71997a8e43, State: Initialized, Role: FOLLOWER
I20260925 14:59:00.601054 27780 consensus_queue.cc:260] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.601331 27780 ts_tablet_manager.cc:1428] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:00.601462 27504 heartbeater.cc:503] Master 127.9.141.126:40647 was elected leader, sending a full tablet report...
I20260925 14:59:00.601436 27782 raft_consensus.cc:348] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.601614 27782 raft_consensus.cc:374] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:00.601660 27782 raft_consensus.cc:729] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c0506cfa4345b49b8f3b1c4cf255f0, State: Initialized, Role: FOLLOWER
I20260925 14:59:00.601797 27782 consensus_queue.cc:260] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.602062 27782 ts_tablet_manager.cc:1428] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 14:59:00.602084 27766 heartbeater.cc:503] Master 127.9.141.126:40647 was elected leader, sending a full tablet report...
I20260925 14:59:00.602475 27781 raft_consensus.cc:348] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.602648 27781 raft_consensus.cc:374] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:00.602699 27781 raft_consensus.cc:729] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25811f96fa2447458af3f38cc171344f, State: Initialized, Role: FOLLOWER
I20260925 14:59:00.602818 27781 consensus_queue.cc:260] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.603132 27781 ts_tablet_manager.cc:1428] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:59:00.603268 27635 heartbeater.cc:503] Master 127.9.141.126:40647 was elected leader, sending a full tablet report...
W20260925 14:59:00.685266 27636 tablet.cc:2369] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:00.807900 27505 tablet.cc:2369] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:00.811932 27767 tablet.cc:2369] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:59:00.819051 27787 raft_consensus.cc:482] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:00.819167 27787 raft_consensus.cc:504] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.819474 27787 leader_election.cc:288] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 098c3ec1270a47cf81d8bc71997a8e43 (127.9.141.65:38445), 25811f96fa2447458af3f38cc171344f (127.9.141.66:34851)
I20260925 14:59:00.822899 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.819730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:38445 (local address 127.9.141.67:40895)
0925 14:59:00.819797 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.819812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.819862 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.820152 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.820154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.820156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:00.820236 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.820238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.821138 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.821139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.822012 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.822015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.822608 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.822610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.822693 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.822707 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.822767 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.822803 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:59:00.822916 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.819888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:40895 (local address 127.9.141.65:38445)
0925 14:59:00.819964 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.819967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.819978 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:59:00.820010 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.820011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.820025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.820124 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:00.820356 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.821039 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.822113 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.822516 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.822577 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.822584 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.822784 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.822830 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:59:00.823029 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.819957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:57315 (local address 127.9.141.66:34851)
0925 14:59:00.820032 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.820034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.820160 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:59:00.820288 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.820289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.820301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.820369 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:00.820562 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.821061 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.822172 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.822588 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.822674 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.822683 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.822902 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.822954 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:59:00.823043 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.819870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:34851 (local address 127.9.141.67:57315)
0925 14:59:00.820071 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.820083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:00.820146 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:00.820413 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:00.820414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:00.820416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:00.820483 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.820485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.821233 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.821234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.822039 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:00.822041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.822697 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.822698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:00.822808 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.822812 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:00.822877 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:59:00.822913 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":140,"threads_started":1}
I20260925 14:59:00.823385 27589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcb6536506e40a9bb6fa1736ba3b7c9" candidate_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25811f96fa2447458af3f38cc171344f" is_pre_election: true
I20260925 14:59:00.823376 27458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcb6536506e40a9bb6fa1736ba3b7c9" candidate_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098c3ec1270a47cf81d8bc71997a8e43" is_pre_election: true
I20260925 14:59:00.823552 27589 raft_consensus.cc:2454] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c0506cfa4345b49b8f3b1c4cf255f0 in term 0.
I20260925 14:59:00.823570 27458 raft_consensus.cc:2454] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c0506cfa4345b49b8f3b1c4cf255f0 in term 0.
I20260925 14:59:00.823819 27654 leader_election.cc:302] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25811f96fa2447458af3f38cc171344f, 33c0506cfa4345b49b8f3b1c4cf255f0; no voters:
I20260925 14:59:00.823951 27787 raft_consensus.cc:2781] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:00.824012 27787 raft_consensus.cc:482] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:00.824039 27787 raft_consensus.cc:3037] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:00.824873 27787 raft_consensus.cc:504] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.825032 27787 leader_election.cc:288] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [CANDIDATE]: Term 1 election: Requested vote from peers 098c3ec1270a47cf81d8bc71997a8e43 (127.9.141.65:38445), 25811f96fa2447458af3f38cc171344f (127.9.141.66:34851)
I20260925 14:59:00.825270 27589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcb6536506e40a9bb6fa1736ba3b7c9" candidate_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25811f96fa2447458af3f38cc171344f"
I20260925 14:59:00.825192 27458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcb6536506e40a9bb6fa1736ba3b7c9" candidate_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "098c3ec1270a47cf81d8bc71997a8e43"
I20260925 14:59:00.825354 27589 raft_consensus.cc:3037] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:00.825356 27458 raft_consensus.cc:3037] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:00.826383 27589 raft_consensus.cc:2454] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c0506cfa4345b49b8f3b1c4cf255f0 in term 1.
I20260925 14:59:00.826383 27458 raft_consensus.cc:2454] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c0506cfa4345b49b8f3b1c4cf255f0 in term 1.
I20260925 14:59:00.826577 27654 leader_election.cc:302] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 098c3ec1270a47cf81d8bc71997a8e43, 33c0506cfa4345b49b8f3b1c4cf255f0; no voters:
I20260925 14:59:00.826726 27787 raft_consensus.cc:2781] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:00.826897 27787 raft_consensus.cc:686] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 1 LEADER]: Becoming Leader. State: Replica: 33c0506cfa4345b49b8f3b1c4cf255f0, State: Running, Role: LEADER
I20260925 14:59:00.827013 27787 consensus_queue.cc:237] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } }
I20260925 14:59:00.828037 27315 catalog_manager.cc:5495] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33c0506cfa4345b49b8f3b1c4cf255f0 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33c0506cfa4345b49b8f3b1c4cf255f0" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 33187 } health_report { overall_health: HEALTHY } } }
I20260925 14:59:00.877256 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.874203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.9.141.67:33187)
0925 14:59:00.874274 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:00.874277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:00.874286 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:00.874502 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:00.874503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:00.874522 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:00.874617 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:00.874786 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.875304 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.876325 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:00.876621 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:00.876706 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:00.876884 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:00.877002 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:00.877101 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:00.877119 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 14:59:00.877138 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:59:00.879287 27458 raft_consensus.cc:1261] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 1 FOLLOWER]: Refusing update from remote peer 33c0506cfa4345b49b8f3b1c4cf255f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:00.879287 27589 raft_consensus.cc:1261] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 1 FOLLOWER]: Refusing update from remote peer 33c0506cfa4345b49b8f3b1c4cf255f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:00.879638 27787 consensus_queue.cc:1035] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "098c3ec1270a47cf81d8bc71997a8e43" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:00.879846 27790 consensus_queue.cc:1035] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25811f96fa2447458af3f38cc171344f" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:00.883661 27795 mvcc.cc:205] Tried to move back new op lower bound from 7333266804238614528 to 7333266804028080128. Current Snapshot: MvccSnapshot[applied={T|T < 7333266804238614528 or (T in {7333266804238614528})}]
I20260925 14:59:00.884727 27316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:00.884809 27316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:00.885558 27316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8643695f3c2a4d3b8acf6dd2f21b1b90: Sending DeleteTablet for 3 replicas of tablet abcb6536506e40a9bb6fa1736ba3b7c9
I20260925 14:59:00.885874 27438 tablet_service.cc:1514] Processing DeleteTablet for tablet abcb6536506e40a9bb6fa1736ba3b7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:00 UTC) from {username='slave'} at 127.0.0.1:34254
I20260925 14:59:00.885892 27700 tablet_service.cc:1514] Processing DeleteTablet for tablet abcb6536506e40a9bb6fa1736ba3b7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:00 UTC) from {username='slave'} at 127.0.0.1:45660
I20260925 14:59:00.886133 27803 tablet_replica.cc:330] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: stopping tablet replica
I20260925 14:59:00.886154 27569 tablet_service.cc:1514] Processing DeleteTablet for tablet abcb6536506e40a9bb6fa1736ba3b7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:00 UTC) from {username='slave'} at 127.0.0.1:38548
I20260925 14:59:00.886246 27803 raft_consensus.cc:2229] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:00.886397 27803 raft_consensus.cc:2258] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:00.886400 27805 tablet_replica.cc:330] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: stopping tablet replica
I20260925 14:59:00.886586 27805 raft_consensus.cc:2229] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:00.886669 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27376
I20260925 14:59:00.886893 27803 ts_tablet_manager.cc:1905] T abcb6536506e40a9bb6fa1736ba3b7c9 P 098c3ec1270a47cf81d8bc71997a8e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:00.887923 27805 raft_consensus.cc:2258] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:00.888490 27805 ts_tablet_manager.cc:1905] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:00.890432 27805 ts_tablet_manager.cc:1918] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:00.890559 27805 log.cc:1196] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/abcb6536506e40a9bb6fa1736ba3b7c9
I20260925 14:59:00.891122 27805 ts_tablet_manager.cc:1939] T abcb6536506e40a9bb6fa1736ba3b7c9 P 25811f96fa2447458af3f38cc171344f: Deleting consensus metadata
I20260925 14:59:00.891700 27301 catalog_manager.cc:4841] TS 25811f96fa2447458af3f38cc171344f (127.9.141.66:34851): tablet abcb6536506e40a9bb6fa1736ba3b7c9 (table test-table [id=d09c8b90cdfe4d3cbea42f39586cc75e]) successfully deleted
I20260925 14:59:00.891759 27804 tablet_replica.cc:330] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: stopping tablet replica
I20260925 14:59:00.891871 27804 raft_consensus.cc:2229] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:00.891984 27804 raft_consensus.cc:2258] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 14:59:00.892587 27301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:38445 (error 108)
I20260925 14:59:00.892596 27804 ts_tablet_manager.cc:1905] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:00.892732 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27507
I20260925 14:59:00.893692 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:00.893573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:38445 (local address 127.0.0.1:34262)
0925 14:59:00.893621 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:00.893658 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:38445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:59:00.893893 27301 catalog_manager.cc:4584] TS 098c3ec1270a47cf81d8bc71997a8e43 (127.9.141.65:38445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abcb6536506e40a9bb6fa1736ba3b7c9: Network error: Client connection negotiation failed: client connection to 127.9.141.65:38445: connect: Connection refused (error 111)
I20260925 14:59:00.894722 27804 ts_tablet_manager.cc:1918] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:00.894852 27804 log.cc:1196] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/abcb6536506e40a9bb6fa1736ba3b7c9
I20260925 14:59:00.895179 27804 ts_tablet_manager.cc:1939] T abcb6536506e40a9bb6fa1736ba3b7c9 P 33c0506cfa4345b49b8f3b1c4cf255f0: Deleting consensus metadata
I20260925 14:59:00.895922 27303 catalog_manager.cc:4841] TS 33c0506cfa4345b49b8f3b1c4cf255f0 (127.9.141.67:33187): tablet abcb6536506e40a9bb6fa1736ba3b7c9 (table test-table [id=d09c8b90cdfe4d3cbea42f39586cc75e]) successfully deleted
I20260925 14:59:00.899323 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27638
I20260925 14:59:00.904843 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27285
2026-09-25T14:59:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (907 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:00 dist-test-slave-1sv7 krb5kdc[27809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:00 dist-test-slave-1sv7 krb5kdc[27809](info): set up 2 sockets
Sep 25 14:59:00 dist-test-slave-1sv7 krb5kdc[27809](info): commencing operation
krb5kdc: starting...
W20260925 14:59:01.976080 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:02.101012 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37643
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:02.182080 27822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:02.182276 27822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:02.182296 27822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:02.183339 27822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:02.183373 27822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:02.183387 27822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:02.183400 27822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:02.183413 27822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:02.184787 27822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:02.184969 27822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:02.185125 27822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:02.189939 27828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.190114 27822 server_base.cc:1034] running on GCE node
W20260925 14:59:02.189944 27830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.189962 27827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.190423 27822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:02.190708 27822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:02.191886 27822 hybrid_clock.cc:648] HybridClock initialized: now 1790348342191864 us; error 40 us; skew 500 ppm
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:02.194439 27822 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:02.194911 27822 webserver.cc:460] Webserver started at http://127.9.141.126:32831/ using document root <none> and password file <none>
I20260925 14:59:02.195132 27822 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:02.195197 27822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:02.195353 27822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:02.196440 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "f50c062cbec047a19a20c08fb958fb30"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "0c601998bb3b1e7bc63c1da0be9b492f"
server_key_iv: "d97cb7eb6820e2c53919b274454c0c25"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.196811 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "f50c062cbec047a19a20c08fb958fb30"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "0c601998bb3b1e7bc63c1da0be9b492f"
server_key_iv: "d97cb7eb6820e2c53919b274454c0c25"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.198129 27822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:02.198998 27837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.199198 27822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:02.199304 27822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "f50c062cbec047a19a20c08fb958fb30"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "0c601998bb3b1e7bc63c1da0be9b492f"
server_key_iv: "d97cb7eb6820e2c53919b274454c0c25"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.199393 27822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:02.225628 27822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:02.228148 27822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:02.228343 27822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:02.232877 27822 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:37643
I20260925 14:59:02.232967 27889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:37643 every 8 connection(s)
I20260925 14:59:02.233364 27822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:02.234416 27890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:02.236429 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27822
I20260925 14:59:02.236580 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:02.236826 9781 external_mini_cluster.cc:1428] Setting key 264a33b291113451ec16378a94b16305
I20260925 14:59:02.237120 27890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: Bootstrap starting.
I20260925 14:59:02.237912 27890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.238219 27890 log.cc:824] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.238919 27890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: No bootstrap required, opened a new log
I20260925 14:59:02.240947 27890 raft_consensus.cc:348] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c062cbec047a19a20c08fb958fb30" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37643 } }
I20260925 14:59:02.241106 27890 raft_consensus.cc:374] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.241154 27890 raft_consensus.cc:729] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50c062cbec047a19a20c08fb958fb30, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.241294 27890 consensus_queue.cc:260] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c062cbec047a19a20c08fb958fb30" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37643 } }
I20260925 14:59:02.241385 27890 raft_consensus.cc:388] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:02.241429 27890 raft_consensus.cc:482] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:02.241478 27890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.242056 27890 raft_consensus.cc:504] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c062cbec047a19a20c08fb958fb30" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37643 } }
I20260925 14:59:02.242166 27890 leader_election.cc:302] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f50c062cbec047a19a20c08fb958fb30; no voters:
I20260925 14:59:02.242331 27890 leader_election.cc:288] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:02.242362 27895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:02.242482 27895 raft_consensus.cc:686] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [term 1 LEADER]: Becoming Leader. State: Replica: f50c062cbec047a19a20c08fb958fb30, State: Running, Role: LEADER
I20260925 14:59:02.242563 27895 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c062cbec047a19a20c08fb958fb30" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37643 } }
I20260925 14:59:02.242599 27890 sys_catalog.cc:564] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:02.242863 27895 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50c062cbec047a19a20c08fb958fb30. Latest consensus state: current_term: 1 leader_uuid: "f50c062cbec047a19a20c08fb958fb30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c062cbec047a19a20c08fb958fb30" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37643 } } }
I20260925 14:59:02.242934 27895 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:02.242880 27896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f50c062cbec047a19a20c08fb958fb30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50c062cbec047a19a20c08fb958fb30" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37643 } } }
I20260925 14:59:02.243106 27896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30 [sys.catalog]: This master's current role is: LEADER
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
W20260925 14:59:02.244638 27910 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:02.245103 27910 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:02.245360 27900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:02.245553 27900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:02.247311 27900 catalog_manager.cc:1347] Generated new cluster ID: 2b610c57d4b347689f2903864a04fe41
I20260925 14:59:02.247366 27900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:02.248617 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.237527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.9.141.126:37643)
0925 14:59:02.237706 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.237714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.237738 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:59:02.238232 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.238239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.238339 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.238482 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.239169 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.240166 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.241034 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.241639 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.241778 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.244782 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.244786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.244786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.244800 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.246921 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.247409 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.247411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.247412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.247491 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.247708 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.247709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.247710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.247978 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.248168 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.248247 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.248408 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260925 14:59:02.253441 27900 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:02.254124 27900 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:02.260512 27900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: Generated new TSK 0
I20260925 14:59:02.260723 27900 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:59:02.296260 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37643
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:02.383556 27916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:02.383761 27916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:02.383785 27916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:02.384990 27916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:02.385032 27916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:02.385059 27916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:59:02.386574 27916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:02.386817 27916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:02.387053 27916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:02.391950 27924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.391955 27921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.391947 27922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.392454 27916 server_base.cc:1034] running on GCE node
I20260925 14:59:02.392648 27916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:02.392848 27916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:02.394016 27916 hybrid_clock.cc:648] HybridClock initialized: now 1790348342394009 us; error 42 us; skew 500 ppm
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:02.396605 27916 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:59:02.397043 27916 webserver.cc:460] Webserver started at http://127.9.141.65:34943/ using document root <none> and password file <none>
I20260925 14:59:02.397276 27916 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:02.397342 27916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:02.397500 27916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:02.398526 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "63e2e76490654b1f8300ec71638be0ff"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "7d5ead807bcab8d33498d7a20cd31b24"
server_key_iv: "079833c1d90bd8010caf78c9da4a00f7"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.398890 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "63e2e76490654b1f8300ec71638be0ff"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "7d5ead807bcab8d33498d7a20cd31b24"
server_key_iv: "079833c1d90bd8010caf78c9da4a00f7"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.400208 27916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:59:02.401015 27931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.401178 27916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:59:02.401278 27916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "63e2e76490654b1f8300ec71638be0ff"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "7d5ead807bcab8d33498d7a20cd31b24"
server_key_iv: "079833c1d90bd8010caf78c9da4a00f7"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.401360 27916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:02.409544 27916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:02.411770 27916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:02.411929 27916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:02.412164 27916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:02.412536 27916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:02.412588 27916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.412629 27916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:02.412657 27916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.418426 27916 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:46649
I20260925 14:59:02.418464 28044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:46649 every 8 connection(s)
I20260925 14:59:02.418804 27916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:02.421908 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 27916
I20260925 14:59:02.422015 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:02.422260 9781 external_mini_cluster.cc:1428] Setting key 577487aa51e092f91eb2fd8826f9310e
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:02.429350 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.419534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:46463 (local address 127.9.141.126:37643)
0925 14:59:02.419616 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.419619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.419811 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:59:02.420371 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.420395 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.420408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.420471 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.421405 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.422098 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.423356 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.423696 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.423813 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.426213 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.426214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.426215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.426222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.427741 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.428283 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.428284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.428284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.428334 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.428686 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.428687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.428687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.428807 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.428860 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.429215 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.429286 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:59:02.429622 28047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.419515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37643 (local address 127.9.141.65:46463)
0925 14:59:02.419676 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.419712 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.420260 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.420551 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.420554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.421103 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:02.421298 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.421303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.422232 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.422233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.423263 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.423266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.423820 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.423821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.423923 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.424583 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:02.424591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:02.426103 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:02.427831 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:02.427840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:02.427842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:02.428197 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:02.428443 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:02.428444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:02.428445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:02.428611 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:02.428877 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:02.428884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:02.429057 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.429192 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.429373 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 14:59:02.430166 28045 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37643
I20260925 14:59:02.430276 28045 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:02.430454 28045 heartbeater.cc:511] Master 127.9.141.126:37643 requested a full tablet report, sending...
I20260925 14:59:02.431007 27854 ts_manager.cc:194] Registered new tserver with Master: 63e2e76490654b1f8300ec71638be0ff (127.9.141.65:46649)
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
I20260925 14:59:02.431665 27854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:46463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:59:02.451357 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37643
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:02.532264 28050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:02.532511 28050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:02.532550 28050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:02.533699 28050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:02.533766 28050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:02.533814 28050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:59:02.535286 28050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:02.535559 28050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:02.535808 28050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:59:02.540966 28050 server_base.cc:1034] running on GCE node
W20260925 14:59:02.540952 28055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.540928 28056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.541055 28058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.541345 28050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:02.541594 28050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:02.542768 28050 hybrid_clock.cc:648] HybridClock initialized: now 1790348342542734 us; error 57 us; skew 500 ppm
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:02.545560 28050 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:59:02.546093 28050 webserver.cc:460] Webserver started at http://127.9.141.66:44231/ using document root <none> and password file <none>
I20260925 14:59:02.546342 28050 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:02.546414 28050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:02.546562 28050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:02.547617 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "f6faab44777e4ee78743fea4d0762c28"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "34952736cec7d2d624215969f927c054"
server_key_iv: "5c077f6e55e7e83fee7fdf97d36b072a"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.547976 28050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "f6faab44777e4ee78743fea4d0762c28"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "34952736cec7d2d624215969f927c054"
server_key_iv: "5c077f6e55e7e83fee7fdf97d36b072a"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.549454 28050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:59:02.550271 28065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.550489 28050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:02.550595 28050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "f6faab44777e4ee78743fea4d0762c28"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "34952736cec7d2d624215969f927c054"
server_key_iv: "5c077f6e55e7e83fee7fdf97d36b072a"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.550671 28050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:02.573875 28050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:02.575891 28050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:02.576062 28050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:02.576314 28050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:02.576644 28050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:02.576701 28050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.576745 28050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:02.576768 28050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.582696 28050 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:38439
I20260925 14:59:02.582755 28178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:38439 every 8 connection(s)
I20260925 14:59:02.583058 28050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:02.586455 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28050
I20260925 14:59:02.586540 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:02.586743 9781 external_mini_cluster.cc:1428] Setting key 1ebf0d1ce4edf8fc0e0b7343d30dea7e
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:02.593808 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.583807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:34571 (local address 127.9.141.126:37643)
0925 14:59:02.583880 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.583883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.584079 (+ 196us) server_negotiation.cc:407] Connection header received
0925 14:59:02.584597 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.584598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.584608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.584669 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.585593 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.586082 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.587413 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.587717 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.587780 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.590626 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.590627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.590627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.590633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.592200 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.592769 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.592770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.592771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.592826 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.593141 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.593142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.593142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.593251 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.593308 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.593660 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.593727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:59:02.594067 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.583779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37643 (local address 127.9.141.66:34571)
0925 14:59:02.583949 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.583992 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.584508 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.584728 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.584731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.585301 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:02.585496 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.585501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.586181 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.586182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.587306 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.587309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.587811 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.587812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.587975 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.588719 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:02.588729 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:02.590524 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:02.592292 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:02.592299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:02.592302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:02.592676 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:02.592899 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:02.592900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:02.592900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:02.593060 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:02.593326 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:02.593333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:02.593512 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.593628 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.593821 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260925 14:59:02.594662 28179 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37643
I20260925 14:59:02.594771 28179 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:02.594980 28179 heartbeater.cc:511] Master 127.9.141.126:37643 requested a full tablet report, sending...
I20260925 14:59:02.595396 27854 ts_manager.cc:194] Registered new tserver with Master: f6faab44777e4ee78743fea4d0762c28 (127.9.141.66:38439)
I20260925 14:59:02.595794 27854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:34571
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:59:02.616211 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37643
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:02.708366 28184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:02.708654 28184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:02.708678 28184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:02.709857 28184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:02.709901 28184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:02.709928 28184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:59:02.711364 28184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:02.711598 28184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:02.711843 28184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:59:02.717084 28184 server_base.cc:1034] running on GCE node
W20260925 14:59:02.717043 28189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.717038 28190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.717088 28192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.717634 28184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:02.717913 28184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:02.719103 28184 hybrid_clock.cc:648] HybridClock initialized: now 1790348342719086 us; error 50 us; skew 500 ppm
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:02.721792 28184 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:59:02.722260 28184 webserver.cc:460] Webserver started at http://127.9.141.67:43511/ using document root <none> and password file <none>
I20260925 14:59:02.722509 28184 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:02.722589 28184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:02.722738 28184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:02.723860 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "98daad5efe0144099e0c017bac3c7f3f"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "2d66f7222c00db728aa81f88d1fdb342"
server_key_iv: "00631b8bc60c29f7d50c92dffa426d1e"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.724224 28184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "98daad5efe0144099e0c017bac3c7f3f"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "2d66f7222c00db728aa81f88d1fdb342"
server_key_iv: "00631b8bc60c29f7d50c92dffa426d1e"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.725718 28184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:02.726578 28199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.726796 28184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:59:02.726897 28184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "98daad5efe0144099e0c017bac3c7f3f"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "2d66f7222c00db728aa81f88d1fdb342"
server_key_iv: "00631b8bc60c29f7d50c92dffa426d1e"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.726981 28184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:02.745245 28184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:02.747284 28184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:02.747448 28184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:02.747690 28184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:02.747989 28184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:02.748054 28184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.748126 28184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:02.748157 28184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.754366 28184 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:46341
I20260925 14:59:02.754410 28312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:46341 every 8 connection(s)
I20260925 14:59:02.754746 28184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:59:02 dist-test-slave-1sv7 krb5kdc[27809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348342, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:02.761489 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28184
I20260925 14:59:02.761588 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:02.761787 9781 external_mini_cluster.cc:1428] Setting key 074cdd08062af158a08235a2fbd79968
I20260925 14:59:02.764801 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.755513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:56687 (local address 127.9.141.126:37643)
0925 14:59:02.755592 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.755594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.755888 (+ 294us) server_negotiation.cc:407] Connection header received
0925 14:59:02.756444 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.756446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.756459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.756528 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.757566 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.758072 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.759120 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.759313 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.759371 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.761907 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.761914 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.761914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.761921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.763414 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.763794 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.763795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.763795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.763843 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.764142 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.764143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.764144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.764272 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.764339 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.764657 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.764733 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:59:02.765086 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.755407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37643 (local address 127.9.141.67:56687)
0925 14:59:02.755714 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.755777 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.756290 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.756607 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.756611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.757253 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:02.757470 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.757476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.758174 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.758175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.759029 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.759033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.759397 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.759398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.759502 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.760081 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:02.760089 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:02.761776 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:02.763489 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:02.763495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:02.763497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:02.763709 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:02.763933 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:02.763934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:02.763934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:02.764045 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:02.764342 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:02.764347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:02.764531 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.764621 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.764772 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
I20260925 14:59:02.765576 28313 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37643
I20260925 14:59:02.765686 28313 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:02.765893 28313 heartbeater.cc:511] Master 127.9.141.126:37643 requested a full tablet report, sending...
I20260925 14:59:02.766289 27854 ts_manager.cc:194] Registered new tserver with Master: 98daad5efe0144099e0c017bac3c7f3f (127.9.141.67:46341)
I20260925 14:59:02.766710 27854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:56687
I20260925 14:59:02.773612 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:02.781498 27893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.774993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.9.141.126:37643)
0925 14:59:02.775123 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.775125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.775219 (+ 94us) server_negotiation.cc:407] Connection header received
0925 14:59:02.775285 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.775286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.775296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.775352 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:02.775915 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.776548 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.777280 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.777486 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.777572 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.778890 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:02.778892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:02.778892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:02.778900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:02.780151 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.780650 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.780651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.780652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.780708 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:02.780945 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:02.780946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:02.780947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:02.781090 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:02.781205 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.781336 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.781399 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 14:59:02.783150 27854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:02.784821 27854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:02.791807 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.787694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.9.141.65:46649)
0925 14:59:02.787883 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.787888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.787925 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:59:02.788003 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.788005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.788020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.788248 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.788471 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.789515 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.790614 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.791341 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.791421 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.791483 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.791674 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.791718 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260925 14:59:02.791848 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.787490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46649 (local address 127.0.0.1:48996)
0925 14:59:02.787868 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.787880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.787937 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.788328 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.788329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.788331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.788402 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.788405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.789609 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.789611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.790531 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.790533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.791444 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.791445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.791544 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.791561 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.791650 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.791732 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":4,"thread_start_us":62,"threads_started":1}
I20260925 14:59:02.791949 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.787519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.9.141.67:46341)
0925 14:59:02.787694 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.787699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.787712 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:59:02.787758 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.787760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.787778 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.787866 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.788139 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.789312 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.790783 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.791466 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.791535 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.791575 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.791817 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.791863 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260925 14:59:02.792213 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.787305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:46341 (local address 127.0.0.1:52602)
0925 14:59:02.787499 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.787543 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.787637 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.787975 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.787978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.787980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.788057 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.788059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.789416 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.789417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.790698 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.790700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.791564 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.791565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.791634 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.791638 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.791710 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.791750 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":74,"thread_start_us":51,"threads_started":1}
I20260925 14:59:02.792810 27979 tablet_service.cc:1467] Processing CreateTablet for tablet 8131cd35dff3448793c0ad4665bd7c71 (DEFAULT_TABLE table=test-table [id=9871c68166dd464dad86e12c73e88fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:02.793009 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.787353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:38439 (local address 127.0.0.1:43828)
0925 14:59:02.787703 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.787714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.787760 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.788034 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.788035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.788037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.788063 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.788064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.789278 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.789279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.790697 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.790700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.792790 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.792791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.792856 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.792861 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.792898 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.792939 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":43,"thread_start_us":91,"threads_started":1}
I20260925 14:59:02.793087 28247 tablet_service.cc:1467] Processing CreateTablet for tablet 8131cd35dff3448793c0ad4665bd7c71 (DEFAULT_TABLE table=test-table [id=9871c68166dd464dad86e12c73e88fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:02.793203 27979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8131cd35dff3448793c0ad4665bd7c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:02.793329 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.787455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.9.141.66:38439)
0925 14:59:02.787883 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.787888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.787905 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:59:02.787945 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.787947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.787962 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.788122 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.788201 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.789169 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.791607 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.792665 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.793103 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.793174 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.793200 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.793250 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":81,"threads_started":1}
I20260925 14:59:02.793416 28247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8131cd35dff3448793c0ad4665bd7c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:02.797071 28331 tablet_bootstrap.cc:492] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Bootstrap starting.
I20260925 14:59:02.797047 28113 tablet_service.cc:1467] Processing CreateTablet for tablet 8131cd35dff3448793c0ad4665bd7c71 (DEFAULT_TABLE table=test-table [id=9871c68166dd464dad86e12c73e88fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:02.797401 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8131cd35dff3448793c0ad4665bd7c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:02.797765 28331 tablet_bootstrap.cc:654] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.798352 28331 log.cc:824] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.798622 28330 tablet_bootstrap.cc:492] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff: Bootstrap starting.
I20260925 14:59:02.799431 28330 tablet_bootstrap.cc:654] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.800014 28330 log.cc:824] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.800618 28331 tablet_bootstrap.cc:492] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: No bootstrap required, opened a new log
I20260925 14:59:02.800700 28331 ts_tablet_manager.cc:1397] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:59:02.800800 28330 tablet_bootstrap.cc:492] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff: No bootstrap required, opened a new log
I20260925 14:59:02.800874 28330 ts_tablet_manager.cc:1397] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:59:02.801126 28334 tablet_bootstrap.cc:492] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Bootstrap starting.
I20260925 14:59:02.801749 28334 tablet_bootstrap.cc:654] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:02.802282 28334 log.cc:824] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:02.802837 28331 raft_consensus.cc:348] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.802870 28330 raft_consensus.cc:348] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.803023 28331 raft_consensus.cc:374] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.803061 28331 raft_consensus.cc:729] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98daad5efe0144099e0c017bac3c7f3f, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.803105 28334 tablet_bootstrap.cc:492] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: No bootstrap required, opened a new log
I20260925 14:59:02.803040 28330 raft_consensus.cc:374] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.803153 28330 raft_consensus.cc:729] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e2e76490654b1f8300ec71638be0ff, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.803165 28334 ts_tablet_manager.cc:1397] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:02.803215 28331 consensus_queue.cc:260] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.803315 28330 consensus_queue.cc:260] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.803499 28331 ts_tablet_manager.cc:1428] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:59:02.803591 28330 ts_tablet_manager.cc:1428] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:02.803653 28313 heartbeater.cc:503] Master 127.9.141.126:37643 was elected leader, sending a full tablet report...
I20260925 14:59:02.803972 28045 heartbeater.cc:503] Master 127.9.141.126:37643 was elected leader, sending a full tablet report...
I20260925 14:59:02.805431 28334 raft_consensus.cc:348] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.805603 28334 raft_consensus.cc:374] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:02.805626 28334 raft_consensus.cc:729] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6faab44777e4ee78743fea4d0762c28, State: Initialized, Role: FOLLOWER
I20260925 14:59:02.805799 28334 consensus_queue.cc:260] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.806103 28334 ts_tablet_manager.cc:1428] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:02.806102 28179 heartbeater.cc:503] Master 127.9.141.126:37643 was elected leader, sending a full tablet report...
I20260925 14:59:02.809110 28338 raft_consensus.cc:482] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:02.809240 28338 raft_consensus.cc:504] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.809629 28338 leader_election.cc:288] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98daad5efe0144099e0c017bac3c7f3f (127.9.141.67:46341), 63e2e76490654b1f8300ec71638be0ff (127.9.141.65:46649)
I20260925 14:59:02.812513 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.809938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:53955 (local address 127.9.141.67:46341)
0925 14:59:02.809982 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.809984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.809993 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:02.810042 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.810043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.810053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.810112 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.810285 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.810791 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.811769 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.812151 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.812238 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.812246 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.812421 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.812454 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:59:02.812515 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.809862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:46341 (local address 127.9.141.66:53955)
0925 14:59:02.809908 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.809923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.809976 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.810145 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.810146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.810148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.810218 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.810220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.810880 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.810881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.811659 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.811661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.812235 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.812236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.812319 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.812322 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.812406 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.812448 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 14:59:02.812875 28267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8131cd35dff3448793c0ad4665bd7c71" candidate_uuid: "f6faab44777e4ee78743fea4d0762c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98daad5efe0144099e0c017bac3c7f3f" is_pre_election: true
I20260925 14:59:02.812973 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.809969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46649 (local address 127.9.141.66:37937)
0925 14:59:02.810103 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.810112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:02.810153 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:02.810344 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:02.810345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:02.810347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:02.810420 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.810421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.811096 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.811097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.811899 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:02.811901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.812713 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.812714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:02.812804 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.812809 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:02.812865 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:59:02.812903 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 14:59:02.813014 28267 raft_consensus.cc:2454] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6faab44777e4ee78743fea4d0762c28 in term 0.
I20260925 14:59:02.813030 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.810175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:37937 (local address 127.9.141.65:46649)
0925 14:59:02.810223 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.810225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.810233 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:02.810254 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.810255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.810265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.810341 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:02.810502 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.810999 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.812005 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.812518 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.812615 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.812625 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.812885 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.812924 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:59:02.813285 28069 leader_election.cc:302] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98daad5efe0144099e0c017bac3c7f3f, f6faab44777e4ee78743fea4d0762c28; no voters:
I20260925 14:59:02.813298 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8131cd35dff3448793c0ad4665bd7c71" candidate_uuid: "f6faab44777e4ee78743fea4d0762c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e2e76490654b1f8300ec71638be0ff" is_pre_election: true
I20260925 14:59:02.813410 27999 raft_consensus.cc:2454] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6faab44777e4ee78743fea4d0762c28 in term 0.
I20260925 14:59:02.813412 28338 raft_consensus.cc:2781] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:02.813506 28338 raft_consensus.cc:482] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:02.813534 28338 raft_consensus.cc:3037] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.814261 28338 raft_consensus.cc:504] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.814392 28338 leader_election.cc:288] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [CANDIDATE]: Term 1 election: Requested vote from peers 98daad5efe0144099e0c017bac3c7f3f (127.9.141.67:46341), 63e2e76490654b1f8300ec71638be0ff (127.9.141.65:46649)
I20260925 14:59:02.814550 28267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8131cd35dff3448793c0ad4665bd7c71" candidate_uuid: "f6faab44777e4ee78743fea4d0762c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98daad5efe0144099e0c017bac3c7f3f"
I20260925 14:59:02.814620 28267 raft_consensus.cc:3037] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.814595 27999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8131cd35dff3448793c0ad4665bd7c71" candidate_uuid: "f6faab44777e4ee78743fea4d0762c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e2e76490654b1f8300ec71638be0ff"
I20260925 14:59:02.814697 27999 raft_consensus.cc:3037] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:02.815393 28267 raft_consensus.cc:2454] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6faab44777e4ee78743fea4d0762c28 in term 1.
I20260925 14:59:02.815476 27999 raft_consensus.cc:2454] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6faab44777e4ee78743fea4d0762c28 in term 1.
I20260925 14:59:02.815558 28069 leader_election.cc:302] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98daad5efe0144099e0c017bac3c7f3f, f6faab44777e4ee78743fea4d0762c28; no voters:
I20260925 14:59:02.815665 28338 raft_consensus.cc:2781] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:02.815812 28338 raft_consensus.cc:686] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 1 LEADER]: Becoming Leader. State: Replica: f6faab44777e4ee78743fea4d0762c28, State: Running, Role: LEADER
I20260925 14:59:02.815907 28338 consensus_queue.cc:237] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } }
I20260925 14:59:02.816654 27854 catalog_manager.cc:5495] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6faab44777e4ee78743fea4d0762c28 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "f6faab44777e4ee78743fea4d0762c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6faab44777e4ee78743fea4d0762c28" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 38439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:02.828783 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.826306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.9.141.66:38439)
0925 14:59:02.826378 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:02.826380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:02.826388 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:02.826420 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:02.826421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:02.826439 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:02.826527 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:02.826646 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.827124 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.827894 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:02.828120 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:02.828195 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:02.828345 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:02.828505 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:02.828645 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:02.828677 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 14:59:02.828687 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:59:02.830756 28267 raft_consensus.cc:1261] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 1 FOLLOWER]: Refusing update from remote peer f6faab44777e4ee78743fea4d0762c28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:02.830773 27999 raft_consensus.cc:1261] T 8131cd35dff3448793c0ad4665bd7c71 P 63e2e76490654b1f8300ec71638be0ff [term 1 FOLLOWER]: Refusing update from remote peer f6faab44777e4ee78743fea4d0762c28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:02.831180 28340 consensus_queue.cc:1035] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98daad5efe0144099e0c017bac3c7f3f" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:02.831300 28338 consensus_queue.cc:1035] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e2e76490654b1f8300ec71638be0ff" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:02.833719 28343 mvcc.cc:205] Tried to move back new op lower bound from 7333266812231725056 to 7333266812174479360. Current Snapshot: MvccSnapshot[applied={T|T < 7333266812231725056}]
I20260925 14:59:02.834147 28345 mvcc.cc:205] Tried to move back new op lower bound from 7333266812231725056 to 7333266812174479360. Current Snapshot: MvccSnapshot[applied={T|T < 7333266812231725056}]
I20260925 14:59:02.836238 27853 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:02.836360 27853 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:02.837190 27853 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f50c062cbec047a19a20c08fb958fb30: Sending DeleteTablet for 3 replicas of tablet 8131cd35dff3448793c0ad4665bd7c71
I20260925 14:59:02.837524 28247 tablet_service.cc:1514] Processing DeleteTablet for tablet 8131cd35dff3448793c0ad4665bd7c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:02 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:52602
I20260925 14:59:02.837849 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27916
I20260925 14:59:02.837864 28113 tablet_service.cc:1514] Processing DeleteTablet for tablet 8131cd35dff3448793c0ad4665bd7c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:02 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:43828
I20260925 14:59:02.838439 28352 tablet_replica.cc:330] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: stopping tablet replica
I20260925 14:59:02.838560 28352 raft_consensus.cc:2229] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:02.838682 28352 raft_consensus.cc:2258] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:02.838867 28353 tablet_replica.cc:330] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: stopping tablet replica
I20260925 14:59:02.838969 28353 raft_consensus.cc:2229] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:02.839076 28353 raft_consensus.cc:2258] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:02.839581 28353 ts_tablet_manager.cc:1905] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:02.841885 28353 ts_tablet_manager.cc:1918] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:02.841972 28353 log.cc:1196] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/8131cd35dff3448793c0ad4665bd7c71
I20260925 14:59:02.842296 28353 ts_tablet_manager.cc:1939] T 8131cd35dff3448793c0ad4665bd7c71 P f6faab44777e4ee78743fea4d0762c28: Deleting consensus metadata
I20260925 14:59:02.842713 27840 catalog_manager.cc:4841] TS f6faab44777e4ee78743fea4d0762c28 (127.9.141.66:38439): tablet 8131cd35dff3448793c0ad4665bd7c71 (table test-table [id=9871c68166dd464dad86e12c73e88fca]) successfully deleted
I20260925 14:59:02.843354 28352 ts_tablet_manager.cc:1905] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:02.845557 28352 ts_tablet_manager.cc:1918] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:02.845654 28352 log.cc:1196] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/8131cd35dff3448793c0ad4665bd7c71
I20260925 14:59:02.845952 28352 ts_tablet_manager.cc:1939] T 8131cd35dff3448793c0ad4665bd7c71 P 98daad5efe0144099e0c017bac3c7f3f: Deleting consensus metadata
I20260925 14:59:02.846531 27841 catalog_manager.cc:4841] TS 98daad5efe0144099e0c017bac3c7f3f (127.9.141.67:46341): tablet 8131cd35dff3448793c0ad4665bd7c71 (table test-table [id=9871c68166dd464dad86e12c73e88fca]) successfully deleted
W20260925 14:59:02.847378 27841 connection.cc:535] client connection to 127.9.141.65:46649 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:59:02.847507 27841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:59:02.848090 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:02.847955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46649 (local address 127.0.0.1:49010)
0925 14:59:02.848010 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:02.848047 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:46649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:59:02.848093 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28050
W20260925 14:59:02.849215 27841 catalog_manager.cc:4584] TS 63e2e76490654b1f8300ec71638be0ff (127.9.141.65:46649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8131cd35dff3448793c0ad4665bd7c71: Network error: Client connection negotiation failed: client connection to 127.9.141.65:46649: connect: Connection refused (error 111)
I20260925 14:59:02.853231 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28184
I20260925 14:59:02.857846 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 27822
2026-09-25T14:59:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1953 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T14:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:02Z Disabled control of system clock
I20260925 14:59:02.887530 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41359
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:41359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:02.970670 28358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:02.970890 28358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:02.970911 28358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:02.972116 28358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:02.972167 28358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:02.972179 28358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:02.972193 28358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:02.972203 28358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:02.973556 28358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:41359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:41359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:02.973748 28358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:02.973985 28358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:02.978916 28364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:02.978943 28366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.978960 28358 server_base.cc:1034] running on GCE node
W20260925 14:59:02.979156 28363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:02.979400 28358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:02.979664 28358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:02.980818 28358 hybrid_clock.cc:648] HybridClock initialized: now 1790348342980802 us; error 31 us; skew 500 ppm
I20260925 14:59:02.982123 28358 webserver.cc:460] Webserver started at http://127.9.141.126:38101/ using document root <none> and password file <none>
I20260925 14:59:02.982335 28358 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:02.982378 28358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:02.982467 28358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:02.983475 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "1f4ecd2c5f46424197ad70763f07a5e7"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "b723789b4fd377c7c130eb5e69683931"
server_key_iv: "d049c1afedc54e2cdf6a8b1c503cdd7f"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.983772 28358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "1f4ecd2c5f46424197ad70763f07a5e7"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "b723789b4fd377c7c130eb5e69683931"
server_key_iv: "d049c1afedc54e2cdf6a8b1c503cdd7f"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.985080 28358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:02.985860 28372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:02.986107 28358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:02.986181 28358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "1f4ecd2c5f46424197ad70763f07a5e7"
format_stamp: "Formatted at 2026-09-25 14:59:02 on dist-test-slave-1sv7"
server_key: "b723789b4fd377c7c130eb5e69683931"
server_key_iv: "d049c1afedc54e2cdf6a8b1c503cdd7f"
server_key_version: "encryptionkey@0"
I20260925 14:59:02.986294 28358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:02.997085 28358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:02.997403 28358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:02.997582 28358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:03.001372 28358 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:41359
I20260925 14:59:03.001415 28424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:41359 every 8 connection(s)
I20260925 14:59:03.001761 28358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:03.002247 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28358
I20260925 14:59:03.002331 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:03.002550 9781 external_mini_cluster.cc:1428] Setting key 9d0952b165f95dedeb1ac1744342131b
I20260925 14:59:03.002954 28425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:03.006484 28425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7: Bootstrap starting.
I20260925 14:59:03.007200 28425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:03.007539 28425 log.cc:824] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:03.008329 28425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7: No bootstrap required, opened a new log
I20260925 14:59:03.008455 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.003116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.9.141.126:41359)
0925 14:59:03.003312 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.003320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.003345 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:59:03.003765 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.003771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.003834 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.003950 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:03.004336 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.005297 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.006402 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.006888 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.006983 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.007730 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:03.007734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:03.007734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:03.007917 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:03.007949 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:03.007960 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:03.008050 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.008125 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.008205 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260925 14:59:03.010293 28425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41359 } }
I20260925 14:59:03.010462 28425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:03.010511 28425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4ecd2c5f46424197ad70763f07a5e7, State: Initialized, Role: FOLLOWER
I20260925 14:59:03.010636 28425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41359 } }
I20260925 14:59:03.010742 28425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:03.010785 28425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:03.010838 28425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:03.011379 28425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41359 } }
I20260925 14:59:03.011495 28425 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4ecd2c5f46424197ad70763f07a5e7; no voters:
I20260925 14:59:03.011691 28425 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:03.011747 28430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:03.011911 28430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4ecd2c5f46424197ad70763f07a5e7, State: Running, Role: LEADER
I20260925 14:59:03.012012 28425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:03.012074 28430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41359 } }
I20260925 14:59:03.012538 28431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41359 } } }
I20260925 14:59:03.012557 28432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4ecd2c5f46424197ad70763f07a5e7. Latest consensus state: current_term: 1 leader_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4ecd2c5f46424197ad70763f07a5e7" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 41359 } } }
I20260925 14:59:03.012635 28431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:03.012692 28432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:03.013453 28439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:03.014081 28439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:03.015377 28439 catalog_manager.cc:1347] Generated new cluster ID: f0b61ca4c8944fb6828b42d330c5dab8
I20260925 14:59:03.015430 28439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:03.043267 28439 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:03.043848 28439 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:03.057475 28439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7: Generated new TSK 0
I20260925 14:59:03.057711 28439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:03.063220 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41359
--builtin_ntp_servers=127.9.141.84:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:03.144855 28449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:03.145057 28449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:03.145078 28449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:03.146314 28449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:03.146360 28449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:03.146386 28449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:59:03.147801 28449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:03.148039 28449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:03.148335 28449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:03.153250 28457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:03.153264 28454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:03.153227 28455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:03.153569 28449 server_base.cc:1034] running on GCE node
I20260925 14:59:03.153789 28449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:03.153980 28449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:03.155185 28449 hybrid_clock.cc:648] HybridClock initialized: now 1790348343155171 us; error 27 us; skew 500 ppm
I20260925 14:59:03.156538 28449 webserver.cc:460] Webserver started at http://127.9.141.65:33599/ using document root <none> and password file <none>
I20260925 14:59:03.156754 28449 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:03.156796 28449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:03.156885 28449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:03.157922 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "bd707a2c0ea5447f86a9d71572f0f29a"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "710fb9a005ba3632d60decb8123d998d"
server_key_iv: "400839a25e3125c00fc641c7fe6a61f9"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.158218 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "bd707a2c0ea5447f86a9d71572f0f29a"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "710fb9a005ba3632d60decb8123d998d"
server_key_iv: "400839a25e3125c00fc641c7fe6a61f9"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.159443 28449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:03.160100 28463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.160290 28449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:59:03.160419 28449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "bd707a2c0ea5447f86a9d71572f0f29a"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "710fb9a005ba3632d60decb8123d998d"
server_key_iv: "400839a25e3125c00fc641c7fe6a61f9"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.160498 28449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:03.170993 28449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:03.171298 28449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:03.171443 28449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:03.171667 28449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:03.171993 28449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:03.172070 28449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.172112 28449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:03.172153 28449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.177939 28449 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:33199
I20260925 14:59:03.177979 28576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:33199 every 8 connection(s)
I20260925 14:59:03.178328 28449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:03.183828 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.179082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:55217 (local address 127.9.141.126:41359)
0925 14:59:03.179128 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.179130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.179418 (+ 288us) server_negotiation.cc:407] Connection header received
0925 14:59:03.179974 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.179975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.179992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.180046 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:03.180850 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.181368 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.182401 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.182603 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.182683 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.183496 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:03.183497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:03.183498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:03.183503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:03.183516 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:03.183521 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:03.183590 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.183701 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.183740 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:59:03.184051 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.179082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41359 (local address 127.9.141.65:55217)
0925 14:59:03.179281 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.179341 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.179855 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.180114 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.180118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.180128 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:03.180740 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.180746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.181491 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.181492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.182281 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.182283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.182693 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.182694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.182825 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.183344 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:03.183345 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:03.183372 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:03.183394 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:03.183592 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:03.183598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:03.183601 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.183704 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.183805 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260925 14:59:03.184643 28577 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41359
I20260925 14:59:03.184723 28577 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:03.184868 28577 heartbeater.cc:511] Master 127.9.141.126:41359 requested a full tablet report, sending...
I20260925 14:59:03.185389 28389 ts_manager.cc:194] Registered new tserver with Master: bd707a2c0ea5447f86a9d71572f0f29a (127.9.141.65:33199)
I20260925 14:59:03.185945 28389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.65:55217
I20260925 14:59:03.188191 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28449
I20260925 14:59:03.188298 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:03.188570 9781 external_mini_cluster.cc:1428] Setting key 5b25938a2f901c18fc27c6923817b3a7
I20260925 14:59:03.189527 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41359
--builtin_ntp_servers=127.9.141.84:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:03.270236 28580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:03.270438 28580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:03.270462 28580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:03.271569 28580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:03.271603 28580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:03.271739 28580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:59:03.273475 28580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:03.273687 28580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:03.273941 28580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:03.279219 28585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:03.279206 28586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:03.279219 28588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:03.279294 28580 server_base.cc:1034] running on GCE node
I20260925 14:59:03.279673 28580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:03.279948 28580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:03.281121 28580 hybrid_clock.cc:648] HybridClock initialized: now 1790348343281098 us; error 57 us; skew 500 ppm
I20260925 14:59:03.282617 28580 webserver.cc:460] Webserver started at http://127.9.141.66:35963/ using document root <none> and password file <none>
I20260925 14:59:03.282902 28580 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:03.282986 28580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:03.283139 28580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:03.284255 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "c635fbde74c17a09085839c104f928cd"
server_key_iv: "b7de7e5ddb5b78264525eeae9632aed8"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.284670 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "c635fbde74c17a09085839c104f928cd"
server_key_iv: "b7de7e5ddb5b78264525eeae9632aed8"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.286024 28580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:03.287089 28594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.287353 28580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:03.287447 28580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "c635fbde74c17a09085839c104f928cd"
server_key_iv: "b7de7e5ddb5b78264525eeae9632aed8"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.287504 28580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:03.304206 28580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:03.304597 28580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:03.304720 28580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:03.304911 28580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:03.305290 28580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:03.305326 28580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.305372 28580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:03.305401 28580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.311113 28580 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:36461
I20260925 14:59:03.311151 28707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:36461 every 8 connection(s)
I20260925 14:59:03.311502 28580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:03.314106 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28580
I20260925 14:59:03.314199 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:03.314424 9781 external_mini_cluster.cc:1428] Setting key ec1fd1f45eeb5023227213eb2ed302e7
I20260925 14:59:03.315383 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41359
--builtin_ntp_servers=127.9.141.84:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:59:03.317392 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.312518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:43597 (local address 127.9.141.126:41359)
0925 14:59:03.312575 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.312577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.312835 (+ 258us) server_negotiation.cc:407] Connection header received
0925 14:59:03.313328 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.313330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.313340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.313408 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:03.314265 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.314841 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.315945 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.316149 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.316230 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.317149 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:03.317151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:03.317151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:03.317155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:03.317169 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:03.317173 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:03.317218 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.317318 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.317338 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:59:03.317718 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.312329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41359 (local address 127.9.141.66:43597)
0925 14:59:03.312636 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.312670 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.313227 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.313464 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.313467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.313475 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:03.314144 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.314151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.314955 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.314956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.315845 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.315848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.316237 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.316238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.316425 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.316987 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:03.316988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:03.317028 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:03.317059 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:03.317237 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:03.317243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:03.317246 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.317324 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.317442 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":122,"threads_started":1}
I20260925 14:59:03.318235 28708 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41359
I20260925 14:59:03.318326 28708 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:03.318519 28708 heartbeater.cc:511] Master 127.9.141.126:41359 requested a full tablet report, sending...
I20260925 14:59:03.318955 28389 ts_manager.cc:194] Registered new tserver with Master: 505f41c3c6f14b6a8c2fd0e5fb6ae62e (127.9.141.66:36461)
I20260925 14:59:03.319526 28389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.66:43597
W20260925 14:59:03.396016 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:03.396209 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:03.396231 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:03.397301 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:03.397337 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:03.397361 28711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:59:03.398588 28711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:41359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:03.398774 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:03.399008 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:03.404093 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:03.404086 28717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:03.404160 28711 server_base.cc:1034] running on GCE node
W20260925 14:59:03.404088 28719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:03.404518 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:03.404726 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:03.405861 28711 hybrid_clock.cc:648] HybridClock initialized: now 1790348343405849 us; error 30 us; skew 500 ppm
I20260925 14:59:03.407042 28711 webserver.cc:460] Webserver started at http://127.9.141.67:34689/ using document root <none> and password file <none>
I20260925 14:59:03.407297 28711 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:03.407344 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:03.407492 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:03.408643 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "bae24d8caa5242f88914fec99e696693"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "3567f560d6527d8d197388d471d018b8"
server_key_iv: "4708f900483d5603a1b07e59bc9ac6d3"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.408998 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "bae24d8caa5242f88914fec99e696693"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "3567f560d6527d8d197388d471d018b8"
server_key_iv: "4708f900483d5603a1b07e59bc9ac6d3"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.410326 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:03.411070 28725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.411242 28711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:59:03.411342 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "bae24d8caa5242f88914fec99e696693"
format_stamp: "Formatted at 2026-09-25 14:59:03 on dist-test-slave-1sv7"
server_key: "3567f560d6527d8d197388d471d018b8"
server_key_iv: "4708f900483d5603a1b07e59bc9ac6d3"
server_key_version: "encryptionkey@0"
I20260925 14:59:03.411458 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:03.454826 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:03.455168 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:03.455338 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:03.455592 28711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:03.455965 28711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:03.456023 28711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.456059 28711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:03.456102 28711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:03.461944 28711 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:39833
I20260925 14:59:03.461985 28838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:39833 every 8 connection(s)
I20260925 14:59:03.462312 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:03.468036 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.463085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:52735 (local address 127.9.141.126:41359)
0925 14:59:03.463195 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.463198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.463483 (+ 285us) server_negotiation.cc:407] Connection header received
0925 14:59:03.464291 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.464292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.464302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.464363 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:03.465159 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.465675 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.466635 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.466826 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.466881 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.467774 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:03.467775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:03.467776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:03.467781 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:03.467796 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:03.467799 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:03.467843 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.467953 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.467974 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 14:59:03.468266 28841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.463078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:41359 (local address 127.9.141.67:52735)
0925 14:59:03.463347 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.463384 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.464142 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.464536 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.464540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.464548 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:03.465055 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.465059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.465759 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.465760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.466551 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.466553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.466902 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.466903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.467010 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.467579 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:03.467579 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:03.467608 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:03.467678 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:03.467863 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:03.467870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:03.467878 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.467936 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.468018 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20260925 14:59:03.468948 28839 heartbeater.cc:347] Connected to a master server at 127.9.141.126:41359
I20260925 14:59:03.469056 28839 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:03.469269 28839 heartbeater.cc:511] Master 127.9.141.126:41359 requested a full tablet report, sending...
I20260925 14:59:03.469679 28389 ts_manager.cc:194] Registered new tserver with Master: bae24d8caa5242f88914fec99e696693 (127.9.141.67:39833)
I20260925 14:59:03.470108 28389 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.67:52735
I20260925 14:59:03.471096 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28711
I20260925 14:59:03.471230 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:03.471489 9781 external_mini_cluster.cc:1428] Setting key 1f4ddf4afc7857a73359a2fe5bfa3292
I20260925 14:59:03.472541 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:03.477030 28428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.473912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.9.141.126:41359)
0925 14:59:03.473985 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.473988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.474105 (+ 117us) server_negotiation.cc:407] Connection header received
0925 14:59:03.474140 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.474140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.474149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.474195 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:03.474354 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.474821 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.475754 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.475932 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.475986 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.476694 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:03.476695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:03.476695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:03.476699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:03.476713 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:03.476716 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:03.476804 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.476910 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.476926 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:59:03.478798 28389 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:03.480356 28389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:03.486866 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.482825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56328 (local address 127.9.141.67:39833)
0925 14:59:03.483135 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.483140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.483150 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:03.483190 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.483192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.483208 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.483304 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.483492 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.484331 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.485977 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.486463 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.486555 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.486649 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.486747 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.486783 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":81,"threads_started":1}
I20260925 14:59:03.486873 28850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.482617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39833 (local address 127.0.0.1:56328)
0925 14:59:03.482943 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.482954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.483013 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.483334 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.483337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.483340 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.483410 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.483412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.484559 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.484561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.485883 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.485885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.486558 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.486559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.486634 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.486647 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.486733 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.486779 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":134,"threads_started":1}
I20260925 14:59:03.487083 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.482562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:36461 (local address 127.0.0.1:38654)
0925 14:59:03.482742 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.482782 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.482870 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.483495 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.483496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.483497 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.483528 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.483529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.484804 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.484804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.485871 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.485874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.486848 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.486849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.486901 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.486905 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.486952 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.486999 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260925 14:59:03.487331 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.482670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.9.141.66:36461)
0925 14:59:03.483161 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.483165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.483174 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:03.483208 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.483210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.483223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.483303 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.483605 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.484702 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.485967 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.486666 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.486726 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.486812 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.487205 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.487252 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":85,"threads_started":1}
I20260925 14:59:03.487342 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.483011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.9.141.65:33199)
0925 14:59:03.483178 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.483182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.483192 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:03.483231 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.483239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.483251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.483353 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.483532 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.484724 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.485774 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.486476 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.486829 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.487050 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.487077 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.487119 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260925 14:59:03.487644 28773 tablet_service.cc:1467] Processing CreateTablet for tablet 526a713282e24b8095175500ef5525da (DEFAULT_TABLE table=test-table [id=e149b14251204e878155f39e6358270c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:03.488056 28773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526a713282e24b8095175500ef5525da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:03.488052 28642 tablet_service.cc:1467] Processing CreateTablet for tablet 526a713282e24b8095175500ef5525da (DEFAULT_TABLE table=test-table [id=e149b14251204e878155f39e6358270c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:03.488476 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.482819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33199 (local address 127.0.0.1:58038)
0925 14:59:03.482993 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.483002 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.483043 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.483418 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.483419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.483420 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.483454 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.483455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.484816 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.484817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.485676 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.485678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.486579 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.486580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.486652 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.486658 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.486698 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.486758 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260925 14:59:03.488468 28642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526a713282e24b8095175500ef5525da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:03.491124 28853 tablet_bootstrap.cc:492] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Bootstrap starting.
I20260925 14:59:03.491279 28854 tablet_bootstrap.cc:492] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Bootstrap starting.
I20260925 14:59:03.491737 28511 tablet_service.cc:1467] Processing CreateTablet for tablet 526a713282e24b8095175500ef5525da (DEFAULT_TABLE table=test-table [id=e149b14251204e878155f39e6358270c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:03.491971 28511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 526a713282e24b8095175500ef5525da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:03.491977 28854 tablet_bootstrap.cc:654] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:03.492143 28853 tablet_bootstrap.cc:654] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:03.492334 28854 log.cc:824] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:03.492534 28853 log.cc:824] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:03.493176 28854 tablet_bootstrap.cc:492] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: No bootstrap required, opened a new log
I20260925 14:59:03.493261 28854 ts_tablet_manager.cc:1397] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:03.493386 28853 tablet_bootstrap.cc:492] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: No bootstrap required, opened a new log
I20260925 14:59:03.493456 28853 ts_tablet_manager.cc:1397] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 14:59:03.495255 28854 raft_consensus.cc:348] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.495455 28854 raft_consensus.cc:374] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:03.495502 28854 raft_consensus.cc:729] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505f41c3c6f14b6a8c2fd0e5fb6ae62e, State: Initialized, Role: FOLLOWER
I20260925 14:59:03.495570 28857 tablet_bootstrap.cc:492] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: Bootstrap starting.
I20260925 14:59:03.495615 28854 consensus_queue.cc:260] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.495893 28854 ts_tablet_manager.cc:1428] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:03.496074 28708 heartbeater.cc:503] Master 127.9.141.126:41359 was elected leader, sending a full tablet report...
I20260925 14:59:03.496187 28853 raft_consensus.cc:348] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.496351 28853 raft_consensus.cc:374] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:03.496419 28853 raft_consensus.cc:729] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae24d8caa5242f88914fec99e696693, State: Initialized, Role: FOLLOWER
I20260925 14:59:03.496430 28857 tablet_bootstrap.cc:654] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:03.496543 28853 consensus_queue.cc:260] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.496824 28853 ts_tablet_manager.cc:1428] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:59:03.496846 28839 heartbeater.cc:503] Master 127.9.141.126:41359 was elected leader, sending a full tablet report...
I20260925 14:59:03.497146 28857 log.cc:824] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:03.497992 28857 tablet_bootstrap.cc:492] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: No bootstrap required, opened a new log
I20260925 14:59:03.498049 28857 ts_tablet_manager.cc:1397] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 14:59:03.499976 28857 raft_consensus.cc:348] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.500103 28857 raft_consensus.cc:374] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:03.500124 28857 raft_consensus.cc:729] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd707a2c0ea5447f86a9d71572f0f29a, State: Initialized, Role: FOLLOWER
I20260925 14:59:03.500262 28857 consensus_queue.cc:260] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.500349 28861 raft_consensus.cc:482] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:03.500458 28861 raft_consensus.cc:504] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.500509 28857 ts_tablet_manager.cc:1428] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:59:03.500517 28577 heartbeater.cc:503] Master 127.9.141.126:41359 was elected leader, sending a full tablet report...
I20260925 14:59:03.501382 28861 leader_election.cc:288] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 505f41c3c6f14b6a8c2fd0e5fb6ae62e (127.9.141.66:36461), bae24d8caa5242f88914fec99e696693 (127.9.141.67:39833)
I20260925 14:59:03.504379 28579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.500943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:36461 (local address 127.9.141.65:50169)
0925 14:59:03.501140 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.501151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.501201 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.501362 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.501364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.501366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.501434 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.501436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.502261 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.502262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.503401 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.503404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.504046 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.504047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.504194 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.504198 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.504249 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.504307 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 14:59:03.504470 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.501001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:50169 (local address 127.9.141.66:36461)
0925 14:59:03.501051 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.501053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.501259 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:59:03.501284 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.501285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.501296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.501348 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.501524 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.502178 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.503487 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.503974 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.504047 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.504056 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.504301 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.504349 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:59:03.504657 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.500961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:39833 (local address 127.9.141.65:45061)
0925 14:59:03.501523 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.501532 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:03.501613 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:03.501744 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:03.501745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:03.501747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:03.501810 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.501811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.502571 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.502571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.503638 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:03.503640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.504327 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.504327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:03.504440 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.504445 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:03.504526 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:59:03.504569 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":95,"threads_started":1}
I20260925 14:59:03.504693 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.501145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:45061 (local address 127.9.141.67:39833)
0925 14:59:03.501200 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.501202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.501580 (+ 378us) server_negotiation.cc:407] Connection header received
0925 14:59:03.501681 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.501682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.501690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.501733 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:03.501908 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.502492 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.503711 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.504223 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.504317 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.504324 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.504582 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.504625 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:59:03.505043 28662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526a713282e24b8095175500ef5525da" candidate_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" is_pre_election: true
I20260925 14:59:03.505044 28793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526a713282e24b8095175500ef5525da" candidate_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae24d8caa5242f88914fec99e696693" is_pre_election: true
I20260925 14:59:03.505177 28793 raft_consensus.cc:2454] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd707a2c0ea5447f86a9d71572f0f29a in term 0.
I20260925 14:59:03.505184 28662 raft_consensus.cc:2454] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd707a2c0ea5447f86a9d71572f0f29a in term 0.
I20260925 14:59:03.505378 28466 leader_election.cc:302] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bae24d8caa5242f88914fec99e696693, bd707a2c0ea5447f86a9d71572f0f29a; no voters:
I20260925 14:59:03.505503 28861 raft_consensus.cc:2781] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:03.505554 28861 raft_consensus.cc:482] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:03.505584 28861 raft_consensus.cc:3037] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:03.506235 28861 raft_consensus.cc:504] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.506379 28861 leader_election.cc:288] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [CANDIDATE]: Term 1 election: Requested vote from peers 505f41c3c6f14b6a8c2fd0e5fb6ae62e (127.9.141.66:36461), bae24d8caa5242f88914fec99e696693 (127.9.141.67:39833)
I20260925 14:59:03.506510 28662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526a713282e24b8095175500ef5525da" candidate_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e"
I20260925 14:59:03.506572 28662 raft_consensus.cc:3037] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:03.506539 28793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "526a713282e24b8095175500ef5525da" candidate_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bae24d8caa5242f88914fec99e696693"
I20260925 14:59:03.506639 28793 raft_consensus.cc:3037] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:03.507212 28662 raft_consensus.cc:2454] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd707a2c0ea5447f86a9d71572f0f29a in term 1.
I20260925 14:59:03.507380 28465 leader_election.cc:302] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 505f41c3c6f14b6a8c2fd0e5fb6ae62e, bd707a2c0ea5447f86a9d71572f0f29a; no voters:
I20260925 14:59:03.507398 28793 raft_consensus.cc:2454] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd707a2c0ea5447f86a9d71572f0f29a in term 1.
I20260925 14:59:03.507479 28861 raft_consensus.cc:2781] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:03.507539 28861 raft_consensus.cc:686] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 1 LEADER]: Becoming Leader. State: Replica: bd707a2c0ea5447f86a9d71572f0f29a, State: Running, Role: LEADER
I20260925 14:59:03.507618 28861 consensus_queue.cc:237] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } }
I20260925 14:59:03.508456 28387 catalog_manager.cc:5495] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a reported cstate change: term changed from 0 to 1, leader changed from <none> to bd707a2c0ea5447f86a9d71572f0f29a (127.9.141.65). New cstate: current_term: 1 leader_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd707a2c0ea5447f86a9d71572f0f29a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 33199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:03.515502 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.512620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.9.141.65:33199)
0925 14:59:03.512665 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:03.512667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:03.512677 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:03.512720 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:03.512722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:03.512743 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:03.512837 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:03.513015 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.513634 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.514596 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:03.514858 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:03.514917 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:03.515155 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:03.515279 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:03.515356 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:03.515423 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:59:03.515432 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:59:03.517319 28793 raft_consensus.cc:1261] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 1 FOLLOWER]: Refusing update from remote peer bd707a2c0ea5447f86a9d71572f0f29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:03.517366 28662 raft_consensus.cc:1261] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 1 FOLLOWER]: Refusing update from remote peer bd707a2c0ea5447f86a9d71572f0f29a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:03.517704 28862 consensus_queue.cc:1035] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "505f41c3c6f14b6a8c2fd0e5fb6ae62e" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 36461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:03.517833 28861 consensus_queue.cc:1035] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bae24d8caa5242f88914fec99e696693" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:03.520643 28866 mvcc.cc:205] Tried to move back new op lower bound from 7333266815044374528 to 7333266815007948800. Current Snapshot: MvccSnapshot[applied={T|T < 7333266815044374528}]
I20260925 14:59:03.523037 28387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:03.523133 28387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:03.523916 28387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f4ecd2c5f46424197ad70763f07a5e7: Sending DeleteTablet for 3 replicas of tablet 526a713282e24b8095175500ef5525da
I20260925 14:59:03.524330 28511 tablet_service.cc:1514] Processing DeleteTablet for tablet 526a713282e24b8095175500ef5525da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:03 UTC) from {username='slave'} at 127.0.0.1:58038
I20260925 14:59:03.524330 28642 tablet_service.cc:1514] Processing DeleteTablet for tablet 526a713282e24b8095175500ef5525da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:03 UTC) from {username='slave'} at 127.0.0.1:38654
I20260925 14:59:03.524345 28773 tablet_service.cc:1514] Processing DeleteTablet for tablet 526a713282e24b8095175500ef5525da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:03 UTC) from {username='slave'} at 127.0.0.1:56328
I20260925 14:59:03.524575 28876 tablet_replica.cc:330] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a: stopping tablet replica
I20260925 14:59:03.524690 28876 raft_consensus.cc:2229] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:03.524853 28876 raft_consensus.cc:2258] T 526a713282e24b8095175500ef5525da P bd707a2c0ea5447f86a9d71572f0f29a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:03.524936 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28449
I20260925 14:59:03.525524 28875 tablet_replica.cc:330] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: stopping tablet replica
I20260925 14:59:03.525626 28875 raft_consensus.cc:2229] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:03.525744 28875 raft_consensus.cc:2258] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:03.525943 28877 tablet_replica.cc:330] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: stopping tablet replica
I20260925 14:59:03.526041 28877 raft_consensus.cc:2229] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:03.526132 28877 raft_consensus.cc:2258] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:03.526538 28875 ts_tablet_manager.cc:1905] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:03.528264 28875 ts_tablet_manager.cc:1918] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:03.528345 28875 log.cc:1196] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/526a713282e24b8095175500ef5525da
I20260925 14:59:03.528304 28877 ts_tablet_manager.cc:1905] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:03.528650 28875 ts_tablet_manager.cc:1939] T 526a713282e24b8095175500ef5525da P 505f41c3c6f14b6a8c2fd0e5fb6ae62e: Deleting consensus metadata
I20260925 14:59:03.529110 28374 catalog_manager.cc:4841] TS 505f41c3c6f14b6a8c2fd0e5fb6ae62e (127.9.141.66:36461): tablet 526a713282e24b8095175500ef5525da (table test-table [id=e149b14251204e878155f39e6358270c]) successfully deleted
I20260925 14:59:03.529974 28877 ts_tablet_manager.cc:1918] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:03.530062 28877 log.cc:1196] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/526a713282e24b8095175500ef5525da
I20260925 14:59:03.530284 28877 ts_tablet_manager.cc:1939] T 526a713282e24b8095175500ef5525da P bae24d8caa5242f88914fec99e696693: Deleting consensus metadata
I20260925 14:59:03.530738 28375 catalog_manager.cc:4841] TS bae24d8caa5242f88914fec99e696693 (127.9.141.67:39833): tablet 526a713282e24b8095175500ef5525da (table test-table [id=e149b14251204e878155f39e6358270c]) successfully deleted
W20260925 14:59:03.530872 28374 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:33199 (error 108)
I20260925 14:59:03.531407 28850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:03.531273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:33199 (local address 127.0.0.1:58056)
0925 14:59:03.531331 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:03.531380 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:33199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:59:03.531430 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28580
W20260925 14:59:03.531554 28374 catalog_manager.cc:4584] TS bd707a2c0ea5447f86a9d71572f0f29a (127.9.141.65:33199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 526a713282e24b8095175500ef5525da: Network error: Client connection negotiation failed: client connection to 127.9.141.65:33199: connect: Connection refused (error 111)
I20260925 14:59:03.536783 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28711
I20260925 14:59:03.541791 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28358
2026-09-25T14:59:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (683 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:03 dist-test-slave-1sv7 krb5kdc[28881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:03 dist-test-slave-1sv7 krb5kdc[28881](info): set up 2 sockets
Sep 25 14:59:03 dist-test-slave-1sv7 krb5kdc[28881](info): commencing operation
krb5kdc: starting...
W20260925 14:59:04.584610 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:04 dist-test-slave-1sv7 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:04.708348 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37471
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:04.790516 28894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:04.790805 28894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:04.790846 28894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:04.792032 28894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:04.792104 28894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:04.792126 28894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:04.792147 28894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:04.792178 28894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:04.793694 28894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:04.793975 28894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:04.794214 28894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:04.799260 28900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:04.799296 28899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:04.799296 28902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:04.799443 28894 server_base.cc:1034] running on GCE node
I20260925 14:59:04.799677 28894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:04.799979 28894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:04.801154 28894 hybrid_clock.cc:648] HybridClock initialized: now 1790348344801133 us; error 44 us; skew 500 ppm
Sep 25 14:59:04 dist-test-slave-1sv7 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:04.803882 28894 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:04.804546 28894 webserver.cc:460] Webserver started at http://127.9.141.126:41107/ using document root <none> and password file <none>
I20260925 14:59:04.804804 28894 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:04.804874 28894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:04.805055 28894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:04.806133 28894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "744318a391fb49f8ba7dcc66406d65c3"
format_stamp: "Formatted at 2026-09-25 14:59:04 on dist-test-slave-1sv7"
server_key: "4a0d931e6e26d4b3d69eb7c4eaa4355d"
server_key_iv: "1bddbb5e543d99f997e48d03d0d1421b"
server_key_version: "encryptionkey@0"
I20260925 14:59:04.806512 28894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "744318a391fb49f8ba7dcc66406d65c3"
format_stamp: "Formatted at 2026-09-25 14:59:04 on dist-test-slave-1sv7"
server_key: "4a0d931e6e26d4b3d69eb7c4eaa4355d"
server_key_iv: "1bddbb5e543d99f997e48d03d0d1421b"
server_key_version: "encryptionkey@0"
I20260925 14:59:04.807914 28894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:04.808878 28909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:04.809170 28894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:59:04.809281 28894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "744318a391fb49f8ba7dcc66406d65c3"
format_stamp: "Formatted at 2026-09-25 14:59:04 on dist-test-slave-1sv7"
server_key: "4a0d931e6e26d4b3d69eb7c4eaa4355d"
server_key_iv: "1bddbb5e543d99f997e48d03d0d1421b"
server_key_version: "encryptionkey@0"
I20260925 14:59:04.809358 28894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:04.816609 28894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:04.818630 28894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:04.818786 28894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:04.822994 28894 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:37471
I20260925 14:59:04.823048 28961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:37471 every 8 connection(s)
I20260925 14:59:04.823429 28894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:04.824327 28962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:04.826772 28962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3: Bootstrap starting.
I20260925 14:59:04.827335 28962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:04.827580 28962 log.cc:824] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:04.828321 28962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3: No bootstrap required, opened a new log
I20260925 14:59:04.829953 28962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744318a391fb49f8ba7dcc66406d65c3" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37471 } }
I20260925 14:59:04.830091 28962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:04.830113 28962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744318a391fb49f8ba7dcc66406d65c3, State: Initialized, Role: FOLLOWER
I20260925 14:59:04.830299 28962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744318a391fb49f8ba7dcc66406d65c3" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37471 } }
I20260925 14:59:04.830382 28962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:04.830405 28962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:04.830433 28962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:04.830915 28962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744318a391fb49f8ba7dcc66406d65c3" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37471 } }
I20260925 14:59:04.830997 28962 leader_election.cc:302] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 744318a391fb49f8ba7dcc66406d65c3; no voters:
I20260925 14:59:04.831117 28962 leader_election.cc:288] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:04.831171 28965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:04.831339 28965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [term 1 LEADER]: Becoming Leader. State: Replica: 744318a391fb49f8ba7dcc66406d65c3, State: Running, Role: LEADER
I20260925 14:59:04.831364 28962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:04.831467 28965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744318a391fb49f8ba7dcc66406d65c3" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37471 } }
I20260925 14:59:04.831904 28967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 744318a391fb49f8ba7dcc66406d65c3. Latest consensus state: current_term: 1 leader_uuid: "744318a391fb49f8ba7dcc66406d65c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744318a391fb49f8ba7dcc66406d65c3" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37471 } } }
I20260925 14:59:04.831984 28967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:04.832054 28966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "744318a391fb49f8ba7dcc66406d65c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "744318a391fb49f8ba7dcc66406d65c3" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37471 } } }
I20260925 14:59:04.832139 28966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:04.832500 28973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:04.833031 28973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:04.833504 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28894
I20260925 14:59:04.833621 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:04.833827 9781 external_mini_cluster.cc:1428] Setting key 6027b934440cfe99fcb49deec08e1f77
Sep 25 14:59:04 dist-test-slave-1sv7 krb5kdc[28881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:04.841100 28973 catalog_manager.cc:1347] Generated new cluster ID: a4e898d9916e42d585f3365004b0012d
I20260925 14:59:04.841166 28973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:04.845158 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:04.834280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.9.141.126:37471)
0925 14:59:04.834461 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:04.834470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:04.834662 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:59:04.834804 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:04.834810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:04.834853 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:04.834963 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:04.835746 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.836733 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.837819 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:04.838235 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:04.838322 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:04.841116 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:04.841119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:04.841120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:04.841133 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:04.843322 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.843804 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.843807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.843808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.843927 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:04.844158 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:04.844158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:04.844159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:04.844505 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:04.844655 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:04.844805 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:59:04.844941 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260925 14:59:04.858801 28973 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:04.859640 28973 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:04.868794 28973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3: Generated new TSK 0
I20260925 14:59:04.869048 28973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.141.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
Entry for principal kudu/127.9.141.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab.
I20260925 14:59:04.901963 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37471
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:04.986264 28988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:04.986529 28988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:04.986572 28988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:04.987882 28988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:04.987959 28988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:04.988011 28988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:59:04.989755 28988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.65.keytab
--principal=kudu/127.9.141.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:04.990054 28988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:04.990324 28988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:04.996227 28993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:04.996233 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:04.996233 28994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:04.996289 28988 server_base.cc:1034] running on GCE node
I20260925 14:59:04.996654 28988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:04.996913 28988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:04.998056 28988 hybrid_clock.cc:648] HybridClock initialized: now 1790348344998046 us; error 29 us; skew 500 ppm
Sep 25 14:59:04 dist-test-slave-1sv7 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:05.000941 28988 init.cc:377] Logged in from keytab as kudu/127.9.141.65@KRBTEST.COM (short username kudu)
I20260925 14:59:05.001479 28988 webserver.cc:460] Webserver started at http://127.9.141.65:34213/ using document root <none> and password file <none>
I20260925 14:59:05.001823 28988 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:05.001909 28988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:05.002067 28988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:05.003261 28988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "0922a4042cdf44d8a3c1514b0c7c57a2"
format_stamp: "Formatted at 2026-09-25 14:59:04 on dist-test-slave-1sv7"
server_key: "0bd0c37c7cc6ef0c07226c6c546c8597"
server_key_iv: "825ecb30b168d997f20b2dd67cd051d3"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.004093 28988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "0922a4042cdf44d8a3c1514b0c7c57a2"
format_stamp: "Formatted at 2026-09-25 14:59:04 on dist-test-slave-1sv7"
server_key: "0bd0c37c7cc6ef0c07226c6c546c8597"
server_key_iv: "825ecb30b168d997f20b2dd67cd051d3"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.005630 28988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 14:59:05.006399 29003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.006569 28988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:59:05.006690 28988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "0922a4042cdf44d8a3c1514b0c7c57a2"
format_stamp: "Formatted at 2026-09-25 14:59:04 on dist-test-slave-1sv7"
server_key: "0bd0c37c7cc6ef0c07226c6c546c8597"
server_key_iv: "825ecb30b168d997f20b2dd67cd051d3"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.006767 28988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:05.039470 28988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:05.042227 28988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:05.042383 28988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:05.042615 28988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:05.042982 28988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:05.043040 28988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.043082 28988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:05.043109 28988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.049228 28988 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:35175
I20260925 14:59:05.049324 29116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:35175 every 8 connection(s)
I20260925 14:59:05.049667 28988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:59:05 dist-test-slave-1sv7 krb5kdc[28881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348344, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.65@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:05.058267 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 28988
I20260925 14:59:05.058398 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:05.058630 9781 external_mini_cluster.cc:1428] Setting key 21fae95656ecc5262d0846467e46afbd
I20260925 14:59:05.059986 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.050484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37471 (local address 127.9.141.65:48175)
0925 14:59:05.050673 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.050715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.051198 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.051447 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.051451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.052030 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:05.052230 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.052235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.052999 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.053000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.053827 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.053830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.054243 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.054247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.054384 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.055051 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:05.055060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:05.056561 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:05.058226 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:05.058235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:05.058238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:05.058614 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:05.058895 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:05.058896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:05.058896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:05.059052 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:05.059425 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:05.059432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:05.059588 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.059689 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.059823 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260925 14:59:05.059990 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.050508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:48175 (local address 127.9.141.126:37471)
0925 14:59:05.050572 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.050574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.050800 (+ 226us) server_negotiation.cc:407] Connection header received
0925 14:59:05.051289 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.051291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.051304 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.051367 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:05.052320 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.052913 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.053906 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.054102 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.054156 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.056650 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:05.056651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:05.056652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:05.056659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:05.058113 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.058737 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.058738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.058739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.058810 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.059176 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.059177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.059177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.059346 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:05.059425 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.059740 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.059848 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:59:05.060941 29117 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37471
I20260925 14:59:05.061049 29117 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:05.061286 29117 heartbeater.cc:511] Master 127.9.141.126:37471 requested a full tablet report, sending...
I20260925 14:59:05.061841 28926 ts_manager.cc:194] Registered new tserver with Master: 0922a4042cdf44d8a3c1514b0c7c57a2 (127.9.141.65:35175)
I20260925 14:59:05.062464 28926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.65@KRBTEST.COM'} at 127.9.141.65:48175
WARNING: no policy specified for kudu/127.9.141.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
Entry for principal kudu/127.9.141.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab.
I20260925 14:59:05.087131 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37471
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:05.169406 29122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:05.169616 29122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:05.169641 29122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:05.170742 29122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:05.170780 29122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:05.170806 29122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:59:05.172313 29122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.66.keytab
--principal=kudu/127.9.141.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:05.172524 29122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:05.172705 29122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:05.177651 29127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:05.177662 29128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:05.177646 29130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:05.178068 29122 server_base.cc:1034] running on GCE node
I20260925 14:59:05.178270 29122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:05.178526 29122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:05.179684 29122 hybrid_clock.cc:648] HybridClock initialized: now 1790348345179674 us; error 33 us; skew 500 ppm
Sep 25 14:59:05 dist-test-slave-1sv7 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:05.182410 29122 init.cc:377] Logged in from keytab as kudu/127.9.141.66@KRBTEST.COM (short username kudu)
I20260925 14:59:05.182904 29122 webserver.cc:460] Webserver started at http://127.9.141.66:43679/ using document root <none> and password file <none>
I20260925 14:59:05.183127 29122 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:05.183172 29122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:05.183297 29122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:05.184326 29122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "73d3e3f54e074db28189ebb4fdc0e007"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "eb36f4dd94f03e99738ea7112a6da965"
server_key_iv: "2c095f4a834ed3fc53139e59350a4eb5"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.184748 29122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "73d3e3f54e074db28189ebb4fdc0e007"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "eb36f4dd94f03e99738ea7112a6da965"
server_key_iv: "2c095f4a834ed3fc53139e59350a4eb5"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.186058 29122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:05.187065 29137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.187315 29122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:05.187397 29122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "73d3e3f54e074db28189ebb4fdc0e007"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "eb36f4dd94f03e99738ea7112a6da965"
server_key_iv: "2c095f4a834ed3fc53139e59350a4eb5"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.187489 29122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:05.197971 29122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:05.201026 29122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:05.201198 29122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:05.201448 29122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:05.201850 29122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:05.201893 29122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.201943 29122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:05.201967 29122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.209863 29122 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:46293
I20260925 14:59:05.209913 29250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:46293 every 8 connection(s)
I20260925 14:59:05.210336 29122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:05.212486 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29122
I20260925 14:59:05.212607 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:05.212860 9781 external_mini_cluster.cc:1428] Setting key c11cdef7beda14b359a48d3b0047834f
Sep 25 14:59:05 dist-test-slave-1sv7 krb5kdc[28881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.66@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.67@KRBTEST.COM; defaulting to no policy
I20260925 14:59:05.223217 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.211176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:36193 (local address 127.9.141.126:37471)
0925 14:59:05.211230 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.211233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.211465 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:59:05.213280 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.213282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.213293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.213324 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:05.214557 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.215359 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.216597 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.216854 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.216961 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.219775 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:05.219776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:05.219777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:05.219785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:05.221406 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.221966 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.221967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.221967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.222033 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.222347 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.222348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.222349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.222529 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:05.222591 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.223024 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.223114 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:59:05.223379 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.211144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37471 (local address 127.9.141.66:36193)
0925 14:59:05.211318 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.211362 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.212998 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.213743 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.213748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.214156 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:05.214446 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.214452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.215468 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.215468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.216479 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.216482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.216969 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.216970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.217116 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.217707 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:05.217717 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:05.219673 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:05.221509 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:05.221517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:05.221520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:05.221885 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:05.222104 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:05.222105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:05.222105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:05.222258 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:05.222594 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:05.222602 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:05.222846 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.222977 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.223143 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260925 14:59:05.223914 29251 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37471
I20260925 14:59:05.224032 29251 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:05.224224 29251 heartbeater.cc:511] Master 127.9.141.126:37471 requested a full tablet report, sending...
I20260925 14:59:05.224649 28926 ts_manager.cc:194] Registered new tserver with Master: 73d3e3f54e074db28189ebb4fdc0e007 (127.9.141.66:46293)
I20260925 14:59:05.225173 28926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.66@KRBTEST.COM'} at 127.9.141.66:36193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
Entry for principal kudu/127.9.141.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab.
I20260925 14:59:05.245318 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37471
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:05.327735 29256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:05.327947 29256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:05.327971 29256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:05.329197 29256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:05.329244 29256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:05.329272 29256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:59:05.330806 29256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.67.keytab
--principal=kudu/127.9.141.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:37471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:05.331071 29256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:05.331333 29256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:05.336496 29262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:05.336561 29264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:05.336717 29256 server_base.cc:1034] running on GCE node
W20260925 14:59:05.336491 29261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:05.336983 29256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:05.337304 29256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:05.338482 29256 hybrid_clock.cc:648] HybridClock initialized: now 1790348345338458 us; error 39 us; skew 500 ppm
Sep 25 14:59:05 dist-test-slave-1sv7 krb5kdc[28881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:05.347992 29256 init.cc:377] Logged in from keytab as kudu/127.9.141.67@KRBTEST.COM (short username kudu)
I20260925 14:59:05.348694 29256 webserver.cc:460] Webserver started at http://127.9.141.67:38343/ using document root <none> and password file <none>
I20260925 14:59:05.348964 29256 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:05.349041 29256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:05.349196 29256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:05.350315 29256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "ba6f58eba6594fed9721c8c8384db843"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "69279457da91a77b8cff9f2ba9070958"
server_key_iv: "d574df292fb423b446ab96cae95fd17f"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.350698 29256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "ba6f58eba6594fed9721c8c8384db843"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "69279457da91a77b8cff9f2ba9070958"
server_key_iv: "d574df292fb423b446ab96cae95fd17f"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.352186 29256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:05.353176 29271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.353398 29256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:05.353498 29256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "ba6f58eba6594fed9721c8c8384db843"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "69279457da91a77b8cff9f2ba9070958"
server_key_iv: "d574df292fb423b446ab96cae95fd17f"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.353586 29256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:05.377547 29256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:05.380012 29256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:05.380183 29256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:05.380519 29256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:05.380851 29256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:05.380908 29256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.380951 29256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:05.380978 29256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.386847 29256 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:40551
I20260925 14:59:05.386891 29384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:40551 every 8 connection(s)
I20260925 14:59:05.387253 29256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:05.391381 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29256
I20260925 14:59:05.391500 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:05.391710 9781 external_mini_cluster.cc:1428] Setting key 430dbe7df0bb8d51a6d5b501832d2372
Sep 25 14:59:05 dist-test-slave-1sv7 krb5kdc[28881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348345, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.67@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:05.398186 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.388009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:58829 (local address 127.9.141.126:37471)
0925 14:59:05.388083 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.388085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.388331 (+ 246us) server_negotiation.cc:407] Connection header received
0925 14:59:05.388895 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.388896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.388907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.388975 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:05.390099 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.390669 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.392115 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.392503 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.392593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.395056 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:05.395058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:05.395059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:05.395067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:05.396722 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.397157 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.397158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.397159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.397224 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.397493 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.397494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.397494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.397618 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:05.397699 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.397996 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.398077 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:59:05.398406 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.387977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37471 (local address 127.9.141.67:58829)
0925 14:59:05.388199 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.388243 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.388784 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.389085 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.389089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.389770 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:05.389973 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.389978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.390771 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.390771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.392006 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.392008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.392600 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.392601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.392771 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.393333 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:05.393343 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:05.394935 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:05.396826 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:05.396835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:05.396838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:05.397073 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:05.397296 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:05.397297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:05.397297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:05.397416 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:05.397711 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:05.397718 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:05.397859 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.397973 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.398146 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260925 14:59:05.398942 29385 heartbeater.cc:347] Connected to a master server at 127.9.141.126:37471
I20260925 14:59:05.399048 29385 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:05.399271 29385 heartbeater.cc:511] Master 127.9.141.126:37471 requested a full tablet report, sending...
I20260925 14:59:05.399726 28926 ts_manager.cc:194] Registered new tserver with Master: ba6f58eba6594fed9721c8c8384db843 (127.9.141.67:40551)
I20260925 14:59:05.400070 28926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.141.67@KRBTEST.COM'} at 127.9.141.67:58829
I20260925 14:59:05.403695 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:05.411758 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.405200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.9.141.126:37471)
0925 14:59:05.405262 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.405266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.405355 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:59:05.405418 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.405419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.405427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.405480 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:05.406041 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.406522 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.407242 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.407421 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.407477 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.408791 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:05.408792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:05.408793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:05.408799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:05.410259 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.410891 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.410893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.410893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.410958 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:05.411183 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:05.411184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:05.411185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:05.411414 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:05.411501 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.411615 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.411676 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:59:05.413583 28926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:05.415242 28926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:05.421923 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.417758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.9.141.66:46293)
0925 14:59:05.418198 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.418203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.418212 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:05.418323 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.418326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.418343 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.418414 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:05.418520 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.419703 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.420984 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.421515 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.421601 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.421659 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.421800 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.421842 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":73,"threads_started":1}
I20260925 14:59:05.421921 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.417639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:46293 (local address 127.0.0.1:41118)
0925 14:59:05.418002 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.418012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.418064 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.418410 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.418411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.418413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:05.418465 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.418467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.419820 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.419821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.420902 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.420904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.421600 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.421600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.421688 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.421707 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.421776 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.421828 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260925 14:59:05.422087 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.417852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:35175 (local address 127.0.0.1:41442)
0925 14:59:05.418252 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.418262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.418314 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.418584 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.418585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.418588 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:05.418629 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.418630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.419492 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.419493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.420785 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.420788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.421800 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.421800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.421850 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.421854 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.421900 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.421934 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":67,"threads_started":1}
I20260925 14:59:05.422274 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.418047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.9.141.65:35175)
0925 14:59:05.418260 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.418265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.418372 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:59:05.418415 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.418417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.418432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.418507 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:05.418696 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.419388 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.420892 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.421649 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.421713 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.421767 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.422108 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.422220 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260925 14:59:05.422694 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.417639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:40551 (local address 127.0.0.1:53806)
0925 14:59:05.417821 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.417859 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.417923 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.418239 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.418243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.418245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:05.418317 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.418319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.420466 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.420467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.421436 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.421439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.422449 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.422450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.422511 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.422515 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.422571 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.422621 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260925 14:59:05.422904 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.417841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.9.141.67:40551)
0925 14:59:05.417988 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.417994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.418004 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:05.418047 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.418049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.418067 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.418154 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:05.418642 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.419706 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.421539 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.422292 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.422359 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.422411 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.422783 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.422831 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260925 14:59:05.423048 29185 tablet_service.cc:1467] Processing CreateTablet for tablet b57ed627f2fa4a5e984b5b02c5c38911 (DEFAULT_TABLE table=test-table [id=ea33cdb9be094b2d9317def85f07727e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:05.423139 29051 tablet_service.cc:1467] Processing CreateTablet for tablet b57ed627f2fa4a5e984b5b02c5c38911 (DEFAULT_TABLE table=test-table [id=ea33cdb9be094b2d9317def85f07727e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:05.423378 29185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57ed627f2fa4a5e984b5b02c5c38911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:05.423476 29051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57ed627f2fa4a5e984b5b02c5c38911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:05.423548 29319 tablet_service.cc:1467] Processing CreateTablet for tablet b57ed627f2fa4a5e984b5b02c5c38911 (DEFAULT_TABLE table=test-table [id=ea33cdb9be094b2d9317def85f07727e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:05.423815 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b57ed627f2fa4a5e984b5b02c5c38911. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:05.426188 29402 tablet_bootstrap.cc:492] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Bootstrap starting.
I20260925 14:59:05.426803 29402 tablet_bootstrap.cc:654] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:05.426929 29403 tablet_bootstrap.cc:492] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: Bootstrap starting.
I20260925 14:59:05.427161 29404 tablet_bootstrap.cc:492] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Bootstrap starting.
I20260925 14:59:05.427568 29402 log.cc:824] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:05.427614 29403 tablet_bootstrap.cc:654] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:05.427757 29404 tablet_bootstrap.cc:654] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:05.428114 29403 log.cc:824] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:05.428261 29404 log.cc:824] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:05.428959 29403 tablet_bootstrap.cc:492] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: No bootstrap required, opened a new log
I20260925 14:59:05.429045 29402 tablet_bootstrap.cc:492] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: No bootstrap required, opened a new log
I20260925 14:59:05.429090 29404 tablet_bootstrap.cc:492] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: No bootstrap required, opened a new log
I20260925 14:59:05.429071 29403 ts_tablet_manager.cc:1397] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:05.429178 29404 ts_tablet_manager.cc:1397] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:05.429188 29402 ts_tablet_manager.cc:1397] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:05.430511 29404 raft_consensus.cc:348] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.430685 29404 raft_consensus.cc:374] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:05.430747 29404 raft_consensus.cc:729] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6f58eba6594fed9721c8c8384db843, State: Initialized, Role: FOLLOWER
I20260925 14:59:05.430867 29404 consensus_queue.cc:260] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.431157 29404 ts_tablet_manager.cc:1428] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:05.431075 29403 raft_consensus.cc:348] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.431075 29402 raft_consensus.cc:348] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.431300 29402 raft_consensus.cc:374] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:05.431301 29403 raft_consensus.cc:374] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:05.431317 29385 heartbeater.cc:503] Master 127.9.141.126:37471 was elected leader, sending a full tablet report...
I20260925 14:59:05.431340 29402 raft_consensus.cc:729] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d3e3f54e074db28189ebb4fdc0e007, State: Initialized, Role: FOLLOWER
I20260925 14:59:05.431340 29403 raft_consensus.cc:729] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0922a4042cdf44d8a3c1514b0c7c57a2, State: Initialized, Role: FOLLOWER
I20260925 14:59:05.431449 29402 consensus_queue.cc:260] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.431449 29403 consensus_queue.cc:260] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.431707 29402 ts_tablet_manager.cc:1428] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:05.431713 29403 ts_tablet_manager.cc:1428] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:59:05.431829 29251 heartbeater.cc:503] Master 127.9.141.126:37471 was elected leader, sending a full tablet report...
I20260925 14:59:05.431849 29117 heartbeater.cc:503] Master 127.9.141.126:37471 was elected leader, sending a full tablet report...
W20260925 14:59:05.461184 29252 tablet.cc:2369] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:05.550498 29118 tablet.cc:2369] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:05.637868 29386 tablet.cc:2369] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:59:05.696275 29409 raft_consensus.cc:482] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:05.696501 29409 raft_consensus.cc:504] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.696900 29409 leader_election.cc:288] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba6f58eba6594fed9721c8c8384db843 (127.9.141.67:40551), 0922a4042cdf44d8a3c1514b0c7c57a2 (127.9.141.65:35175)
I20260925 14:59:05.700408 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.697085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:40551 (local address 127.9.141.66:46037)
0925 14:59:05.697133 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.697145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.697192 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.697607 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.697608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.697610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:05.697684 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.697686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.698629 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.698630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.699447 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.699450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.700027 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.700028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.700117 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.700126 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.700267 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.700311 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:59:05.700408 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.697377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:46037 (local address 127.9.141.67:40551)
0925 14:59:05.697441 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.697444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.697452 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:05.697483 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.697484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.697495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.697553 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:05.697772 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.698523 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.699546 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.699924 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.700018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.700028 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.700282 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.700318 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:59:05.700812 29339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57ed627f2fa4a5e984b5b02c5c38911" candidate_uuid: "73d3e3f54e074db28189ebb4fdc0e007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6f58eba6594fed9721c8c8384db843" is_pre_election: true
I20260925 14:59:05.700990 29339 raft_consensus.cc:2454] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d3e3f54e074db28189ebb4fdc0e007 in term 0.
I20260925 14:59:05.701244 29139 leader_election.cc:302] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d3e3f54e074db28189ebb4fdc0e007, ba6f58eba6594fed9721c8c8384db843; no voters:
I20260925 14:59:05.701345 29401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.697283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:49971 (local address 127.9.141.65:35175)
0925 14:59:05.697331 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.697333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.697605 (+ 272us) server_negotiation.cc:407] Connection header received
0925 14:59:05.697660 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.697662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.697677 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.697741 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:05.697845 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.698529 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.700364 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.700933 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.700995 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.701006 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.701233 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.701278 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:59:05.701421 29409 raft_consensus.cc:2781] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:05.701505 29409 raft_consensus.cc:482] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:05.701553 29409 raft_consensus.cc:3037] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:05.701623 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.697085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:35175 (local address 127.9.141.66:49971)
0925 14:59:05.697535 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.697544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:05.697612 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:05.697742 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:05.697743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:05.697744 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:05.697804 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.697806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.699549 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.699549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.700225 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:05.700226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.701071 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.701071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:05.701145 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.701150 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:05.701454 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 14:59:05.701500 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":332,"thread_start_us":80,"threads_started":1}
I20260925 14:59:05.701946 29071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57ed627f2fa4a5e984b5b02c5c38911" candidate_uuid: "73d3e3f54e074db28189ebb4fdc0e007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" is_pre_election: true
I20260925 14:59:05.702100 29071 raft_consensus.cc:2454] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d3e3f54e074db28189ebb4fdc0e007 in term 0.
I20260925 14:59:05.702498 29409 raft_consensus.cc:504] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.702649 29409 leader_election.cc:288] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [CANDIDATE]: Term 1 election: Requested vote from peers ba6f58eba6594fed9721c8c8384db843 (127.9.141.67:40551), 0922a4042cdf44d8a3c1514b0c7c57a2 (127.9.141.65:35175)
I20260925 14:59:05.702775 29339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57ed627f2fa4a5e984b5b02c5c38911" candidate_uuid: "73d3e3f54e074db28189ebb4fdc0e007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6f58eba6594fed9721c8c8384db843"
I20260925 14:59:05.702790 29071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b57ed627f2fa4a5e984b5b02c5c38911" candidate_uuid: "73d3e3f54e074db28189ebb4fdc0e007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2"
I20260925 14:59:05.702853 29339 raft_consensus.cc:3037] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:05.702853 29071 raft_consensus.cc:3037] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:05.703704 29339 raft_consensus.cc:2454] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d3e3f54e074db28189ebb4fdc0e007 in term 1.
I20260925 14:59:05.703704 29071 raft_consensus.cc:2454] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d3e3f54e074db28189ebb4fdc0e007 in term 1.
I20260925 14:59:05.703936 29140 leader_election.cc:302] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0922a4042cdf44d8a3c1514b0c7c57a2, 73d3e3f54e074db28189ebb4fdc0e007; no voters:
I20260925 14:59:05.704032 29409 raft_consensus.cc:2781] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:05.704206 29409 raft_consensus.cc:686] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 1 LEADER]: Becoming Leader. State: Replica: 73d3e3f54e074db28189ebb4fdc0e007, State: Running, Role: LEADER
I20260925 14:59:05.704312 29409 consensus_queue.cc:237] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } }
I20260925 14:59:05.705233 28926 catalog_manager.cc:5495] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d3e3f54e074db28189ebb4fdc0e007 (127.9.141.66). New cstate: current_term: 1 leader_uuid: "73d3e3f54e074db28189ebb4fdc0e007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d3e3f54e074db28189ebb4fdc0e007" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 46293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:05.775280 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.772905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.9.141.66:46293)
0925 14:59:05.772948 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.772950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.772958 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:05.773000 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.773002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.773024 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.773087 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:05.773233 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.773701 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.774466 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.774742 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.774797 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.774945 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:05.775044 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:05.775106 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.775189 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.775206 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:05.777227 29339 raft_consensus.cc:1261] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 1 FOLLOWER]: Refusing update from remote peer 73d3e3f54e074db28189ebb4fdc0e007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:05.777252 29071 raft_consensus.cc:1261] T b57ed627f2fa4a5e984b5b02c5c38911 P 0922a4042cdf44d8a3c1514b0c7c57a2 [term 1 FOLLOWER]: Refusing update from remote peer 73d3e3f54e074db28189ebb4fdc0e007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:05.777644 29409 consensus_queue.cc:1035] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6f58eba6594fed9721c8c8384db843" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 40551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:05.777798 29412 consensus_queue.cc:1035] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0922a4042cdf44d8a3c1514b0c7c57a2" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 35175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:05.780678 29417 mvcc.cc:205] Tried to move back new op lower bound from 7333266824300638208 to 7333266824005836800. Current Snapshot: MvccSnapshot[applied={T|T < 7333266824300638208 or (T in {7333266824300638208})}]
I20260925 14:59:05.781665 29415 mvcc.cc:205] Tried to move back new op lower bound from 7333266824300638208 to 7333266824005836800. Current Snapshot: MvccSnapshot[applied={T|T < 7333266824300638208 or (T in {7333266824300638208})}]
I20260925 14:59:05.782438 28925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:05.782546 28925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:05.783303 28925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 744318a391fb49f8ba7dcc66406d65c3: Sending DeleteTablet for 3 replicas of tablet b57ed627f2fa4a5e984b5b02c5c38911
I20260925 14:59:05.783631 29319 tablet_service.cc:1514] Processing DeleteTablet for tablet b57ed627f2fa4a5e984b5b02c5c38911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:05 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:53806
I20260925 14:59:05.783646 29051 tablet_service.cc:1514] Processing DeleteTablet for tablet b57ed627f2fa4a5e984b5b02c5c38911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:05 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:41442
I20260925 14:59:05.783658 29185 tablet_service.cc:1514] Processing DeleteTablet for tablet b57ed627f2fa4a5e984b5b02c5c38911 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:05 UTC) from {username='kudu', principal='kudu/127.9.141.126@KRBTEST.COM'} at 127.0.0.1:41118
I20260925 14:59:05.783905 29427 tablet_replica.cc:330] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: stopping tablet replica
I20260925 14:59:05.783912 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28988
I20260925 14:59:05.783998 29427 raft_consensus.cc:2229] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:05.784085 29427 raft_consensus.cc:2258] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:05.784540 29427 ts_tablet_manager.cc:1905] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:05.786190 29427 ts_tablet_manager.cc:1918] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:05.786267 29427 log.cc:1196] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-1/wal/wals/b57ed627f2fa4a5e984b5b02c5c38911
I20260925 14:59:05.786523 29427 ts_tablet_manager.cc:1939] T b57ed627f2fa4a5e984b5b02c5c38911 P 73d3e3f54e074db28189ebb4fdc0e007: Deleting consensus metadata
I20260925 14:59:05.786921 28912 catalog_manager.cc:4841] TS 73d3e3f54e074db28189ebb4fdc0e007 (127.9.141.66:46293): tablet b57ed627f2fa4a5e984b5b02c5c38911 (table test-table [id=ea33cdb9be094b2d9317def85f07727e]) successfully deleted
I20260925 14:59:05.787339 29425 tablet_replica.cc:330] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: stopping tablet replica
I20260925 14:59:05.787467 29425 raft_consensus.cc:2229] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:05.787590 29425 raft_consensus.cc:2258] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:05.787977 29425 ts_tablet_manager.cc:1905] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:05.789520 29425 ts_tablet_manager.cc:1918] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:05.789662 29425 log.cc:1196] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/b57ed627f2fa4a5e984b5b02c5c38911
I20260925 14:59:05.789927 29425 ts_tablet_manager.cc:1939] T b57ed627f2fa4a5e984b5b02c5c38911 P ba6f58eba6594fed9721c8c8384db843: Deleting consensus metadata
I20260925 14:59:05.790380 28911 catalog_manager.cc:4841] TS ba6f58eba6594fed9721c8c8384db843 (127.9.141.67:40551): tablet b57ed627f2fa4a5e984b5b02c5c38911 (table test-table [id=ea33cdb9be094b2d9317def85f07727e]) successfully deleted
W20260925 14:59:05.799974 28912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:35175 (error 108)
I20260925 14:59:05.800630 29396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.800475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:35175 (local address 127.0.0.1:41448)
0925 14:59:05.800538 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:05.800591 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:35175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:59:05.800624 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29122
W20260925 14:59:05.802357 28912 catalog_manager.cc:4584] TS 0922a4042cdf44d8a3c1514b0c7c57a2 (127.9.141.65:35175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b57ed627f2fa4a5e984b5b02c5c38911: Network error: Client connection negotiation failed: client connection to 127.9.141.65:35175: connect: Connection refused (error 111)
I20260925 14:59:05.805986 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29256
I20260925 14:59:05.810592 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 28894
2026-09-25T14:59:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2268 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T14:59:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:05Z Disabled control of system clock
I20260925 14:59:05.838001 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44951
--webserver_interface=127.9.141.126
--webserver_port=0
--builtin_ntp_servers=127.9.141.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:44951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:05.919888 29432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:05.920222 29432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:05.920260 29432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:05.921705 29432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:05.921756 29432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:05.921769 29432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:05.921782 29432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:05.921797 29432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:05.923209 29432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:44951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:44951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:05.923487 29432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:05.923810 29432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:05.928879 29440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:05.928886 29438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:05.928903 29437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:05.929582 29432 server_base.cc:1034] running on GCE node
I20260925 14:59:05.929873 29432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:05.930203 29432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:05.931368 29432 hybrid_clock.cc:648] HybridClock initialized: now 1790348345931345 us; error 42 us; skew 500 ppm
I20260925 14:59:05.932927 29432 webserver.cc:460] Webserver started at http://127.9.141.126:35615/ using document root <none> and password file <none>
I20260925 14:59:05.933359 29432 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:05.933444 29432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:05.933609 29432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:05.934718 29432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "83ca11965ad74f4e995f85d764d23cea"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "08123b140967baa4ef0cc64b48a797b7"
server_key_iv: "4ad5e990de6cf3694f2a4977eb631bcc"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.935113 29432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "83ca11965ad74f4e995f85d764d23cea"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "08123b140967baa4ef0cc64b48a797b7"
server_key_iv: "4ad5e990de6cf3694f2a4977eb631bcc"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.936826 29432 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:05.937628 29446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:05.937865 29432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:05.937963 29432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "83ca11965ad74f4e995f85d764d23cea"
format_stamp: "Formatted at 2026-09-25 14:59:05 on dist-test-slave-1sv7"
server_key: "08123b140967baa4ef0cc64b48a797b7"
server_key_iv: "4ad5e990de6cf3694f2a4977eb631bcc"
server_key_version: "encryptionkey@0"
I20260925 14:59:05.938016 29432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:05.952185 29432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:05.952646 29432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:05.952832 29432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:05.957808 29432 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:44951
I20260925 14:59:05.957841 29498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:44951 every 8 connection(s)
I20260925 14:59:05.958237 29432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:05.959380 29499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:05.962064 29499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea: Bootstrap starting.
I20260925 14:59:05.962587 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29432
I20260925 14:59:05.962723 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:05.962769 29499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:05.962956 9781 external_mini_cluster.cc:1428] Setting key 2238113e234d908ec526ec61628dbd9d
I20260925 14:59:05.963173 29499 log.cc:824] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:05.964145 29499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea: No bootstrap required, opened a new log
I20260925 14:59:05.967057 29499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ca11965ad74f4e995f85d764d23cea" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44951 } }
I20260925 14:59:05.967280 29499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:05.967321 29499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ca11965ad74f4e995f85d764d23cea, State: Initialized, Role: FOLLOWER
I20260925 14:59:05.967489 29499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ca11965ad74f4e995f85d764d23cea" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44951 } }
I20260925 14:59:05.967594 29499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:05.967620 29499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:05.967693 29499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:05.968418 29499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ca11965ad74f4e995f85d764d23cea" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44951 } }
I20260925 14:59:05.968541 29499 leader_election.cc:302] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ca11965ad74f4e995f85d764d23cea; no voters:
I20260925 14:59:05.969123 29499 leader_election.cc:288] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:05.969213 29504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:05.969372 29504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [term 1 LEADER]: Becoming Leader. State: Replica: 83ca11965ad74f4e995f85d764d23cea, State: Running, Role: LEADER
I20260925 14:59:05.969456 29499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:05.969475 29504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ca11965ad74f4e995f85d764d23cea" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44951 } }
I20260925 14:59:05.969851 29504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ca11965ad74f4e995f85d764d23cea. Latest consensus state: current_term: 1 leader_uuid: "83ca11965ad74f4e995f85d764d23cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ca11965ad74f4e995f85d764d23cea" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44951 } } }
I20260925 14:59:05.969944 29504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:05.969862 29505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ca11965ad74f4e995f85d764d23cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ca11965ad74f4e995f85d764d23cea" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 44951 } } }
I20260925 14:59:05.970106 29505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:05.970278 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:05.963590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.9.141.126:44951)
0925 14:59:05.963811 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:05.963819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:05.963845 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:59:05.964309 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:05.964318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:05.964445 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:05.964587 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:05.964927 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.965832 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.967448 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:05.968176 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:05.968297 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:05.969198 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:05.969203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:05.969203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:05.969447 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:05.969492 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:05.969508 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:05.969873 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:05.969909 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:59:05.970003 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260925 14:59:05.970419 29510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:05.970934 29510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:05.972618 29510 catalog_manager.cc:1347] Generated new cluster ID: 69deb1b2adff43f09108c57e5514308c
I20260925 14:59:05.972678 29510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:06.004209 29510 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:06.004989 29510 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:06.015314 29510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea: Generated new TSK 0
I20260925 14:59:06.015558 29510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:06.020073 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.65:0
--local_ip_for_outbound_sockets=127.9.141.65
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44951
--builtin_ntp_servers=127.9.141.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:06.104028 29523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:06.104300 29523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:06.104341 29523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:06.105700 29523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:06.105777 29523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:06.105822 29523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.65
I20260925 14:59:06.107290 29523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.141.65
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.65
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:06.107574 29523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:06.107899 29523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:06.113091 29528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:06.113091 29529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:06.113142 29531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:06.113340 29523 server_base.cc:1034] running on GCE node
I20260925 14:59:06.113719 29523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:06.113987 29523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:06.115231 29523 hybrid_clock.cc:648] HybridClock initialized: now 1790348346115205 us; error 32 us; skew 500 ppm
I20260925 14:59:06.116870 29523 webserver.cc:460] Webserver started at http://127.9.141.65:40799/ using document root <none> and password file <none>
I20260925 14:59:06.117153 29523 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:06.117203 29523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:06.117368 29523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:06.118501 29523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data/instance:
uuid: "dc8fd68cee1941bd80aaa1fa7604529a"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "84a99b202cbf3432faa3f6a08ddebacb"
server_key_iv: "6700635b94410ad795401b8ffdb2eecf"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.118896 29523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance:
uuid: "dc8fd68cee1941bd80aaa1fa7604529a"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "84a99b202cbf3432faa3f6a08ddebacb"
server_key_iv: "6700635b94410ad795401b8ffdb2eecf"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.120226 29523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:06.121052 29537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.121325 29523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:06.121400 29523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal
uuid: "dc8fd68cee1941bd80aaa1fa7604529a"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "84a99b202cbf3432faa3f6a08ddebacb"
server_key_iv: "6700635b94410ad795401b8ffdb2eecf"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.121493 29523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:06.137401 29523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:06.137722 29523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:06.137871 29523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:06.138103 29523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:06.138437 29523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:06.138489 29523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.138530 29523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:06.138558 29523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.144356 29523 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.65:46423
I20260925 14:59:06.144443 29650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.65:46423 every 8 connection(s)
I20260925 14:59:06.144771 29523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:06.145190 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29523
I20260925 14:59:06.145305 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:06.145550 9781 external_mini_cluster.cc:1428] Setting key ae83b10a06951e18d089dc8aa7f490e1
I20260925 14:59:06.146500 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.66:0
--local_ip_for_outbound_sockets=127.9.141.66
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44951
--builtin_ntp_servers=127.9.141.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:59:06.150689 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.145565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.65:44063 (local address 127.9.141.126:44951)
0925 14:59:06.145640 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.145643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.145919 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:59:06.146524 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.146525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.146537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.146588 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:06.147277 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.148034 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.149125 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.149445 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.149524 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.150381 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:06.150383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:06.150383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:06.150388 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:06.150406 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:06.150411 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:06.150482 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.150533 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.150560 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:59:06.150828 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.145541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44951 (local address 127.9.141.65:44063)
0925 14:59:06.145781 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.145827 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.146420 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.146683 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.146686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.146694 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:06.147186 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.147189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.148150 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.148150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.149037 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.149039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.149555 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.149556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.149673 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.150237 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:06.150237 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:06.150266 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:06.150289 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:06.150481 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:06.150483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:06.150485 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.150533 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.150608 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260925 14:59:06.151484 29651 heartbeater.cc:347] Connected to a master server at 127.9.141.126:44951
I20260925 14:59:06.151592 29651 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:06.151819 29651 heartbeater.cc:511] Master 127.9.141.126:44951 requested a full tablet report, sending...
I20260925 14:59:06.152412 29463 ts_manager.cc:194] Registered new tserver with Master: dc8fd68cee1941bd80aaa1fa7604529a (127.9.141.65:46423)
I20260925 14:59:06.153031 29463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.65:44063
W20260925 14:59:06.228217 29654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:06.228446 29654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:06.228469 29654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:06.229632 29654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:06.229672 29654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:06.229696 29654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.66
I20260925 14:59:06.231098 29654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.141.66
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.66
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:06.231325 29654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:06.231619 29654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:06.236716 29662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:06.236716 29660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:06.236763 29654 server_base.cc:1034] running on GCE node
W20260925 14:59:06.236722 29659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:06.237136 29654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:06.237386 29654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:06.238560 29654 hybrid_clock.cc:648] HybridClock initialized: now 1790348346238554 us; error 37 us; skew 500 ppm
I20260925 14:59:06.239924 29654 webserver.cc:460] Webserver started at http://127.9.141.66:42481/ using document root <none> and password file <none>
I20260925 14:59:06.240172 29654 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:06.240244 29654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:06.240451 29654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:06.241670 29654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data/instance:
uuid: "f619d68800b949f3a079069fb8e01122"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "4c250b441f88428edaadce63ffee8529"
server_key_iv: "9b84c863c3261016fd524df0e621f4f8"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.242025 29654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance:
uuid: "f619d68800b949f3a079069fb8e01122"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "4c250b441f88428edaadce63ffee8529"
server_key_iv: "9b84c863c3261016fd524df0e621f4f8"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.243363 29654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:06.244115 29668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.244292 29654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:59:06.244410 29654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal
uuid: "f619d68800b949f3a079069fb8e01122"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "4c250b441f88428edaadce63ffee8529"
server_key_iv: "9b84c863c3261016fd524df0e621f4f8"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.244499 29654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:06.298659 29654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:06.299047 29654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:06.299225 29654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:06.299492 29654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:06.299844 29654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:06.299898 29654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.299939 29654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:06.299968 29654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.305790 29654 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.66:39937
I20260925 14:59:06.305840 29781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.66:39937 every 8 connection(s)
I20260925 14:59:06.306183 29654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:06.311460 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.307017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.66:57061 (local address 127.9.141.126:44951)
0925 14:59:06.307203 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.307205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.307293 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:59:06.307816 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.307817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.307826 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.307887 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:06.308685 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.309181 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.310231 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.310414 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.310470 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.311232 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:06.311233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:06.311233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:06.311237 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:06.311251 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:06.311255 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:06.311299 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.311379 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.311400 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260925 14:59:06.311708 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.306950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44951 (local address 127.9.141.66:57061)
0925 14:59:06.307158 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.307212 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.307725 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.307953 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.307957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.307966 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:06.308601 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.308608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.309306 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.309307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.310148 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.310150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.310503 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.310504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.310619 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.311101 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:06.311102 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:06.311130 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:06.311152 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:06.311310 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:06.311314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:06.311316 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.311368 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.311447 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260925 14:59:06.312238 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29654
I20260925 14:59:06.312312 29782 heartbeater.cc:347] Connected to a master server at 127.9.141.126:44951
I20260925 14:59:06.312367 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:06.312429 29782 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:06.312624 9781 external_mini_cluster.cc:1428] Setting key 660f216e35a268a4f087e449d5c4af03
I20260925 14:59:06.312664 29782 heartbeater.cc:511] Master 127.9.141.126:44951 requested a full tablet report, sending...
I20260925 14:59:06.313016 29463 ts_manager.cc:194] Registered new tserver with Master: f619d68800b949f3a079069fb8e01122 (127.9.141.66:39937)
I20260925 14:59:06.313473 29463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.66:57061
I20260925 14:59:06.313840 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.141.67:0
--local_ip_for_outbound_sockets=127.9.141.67
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44951
--builtin_ntp_servers=127.9.141.84:46879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:59:06.397811 29785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:06.398041 29785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:06.398074 29785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:06.399186 29785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:06.399247 29785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:06.399290 29785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.141.67
I20260925 14:59:06.400665 29785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:46879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.141.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.141.67
--webserver_port=0
--tserver_master_addrs=127.9.141.126:44951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.141.67
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:06.400921 29785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:06.401211 29785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:06.406176 29790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:06.406225 29791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:06.406207 29785 server_base.cc:1034] running on GCE node
W20260925 14:59:06.406203 29793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:06.406589 29785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:06.406831 29785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:06.408004 29785 hybrid_clock.cc:648] HybridClock initialized: now 1790348346408003 us; error 41 us; skew 500 ppm
I20260925 14:59:06.409286 29785 webserver.cc:460] Webserver started at http://127.9.141.67:35965/ using document root <none> and password file <none>
I20260925 14:59:06.409518 29785 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:06.409585 29785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:06.409739 29785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:06.410768 29785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data/instance:
uuid: "26ee293068004c2f92772c1c267d7189"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "7e0a2d71258e5a757a60fbecb0c2c958"
server_key_iv: "bca932556e7e4a4f77ceb8315b426d8e"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.411121 29785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance:
uuid: "26ee293068004c2f92772c1c267d7189"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "7e0a2d71258e5a757a60fbecb0c2c958"
server_key_iv: "bca932556e7e4a4f77ceb8315b426d8e"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.412525 29785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:59:06.413269 29799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.413447 29785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.413540 29785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal
uuid: "26ee293068004c2f92772c1c267d7189"
format_stamp: "Formatted at 2026-09-25 14:59:06 on dist-test-slave-1sv7"
server_key: "7e0a2d71258e5a757a60fbecb0c2c958"
server_key_iv: "bca932556e7e4a4f77ceb8315b426d8e"
server_key_version: "encryptionkey@0"
I20260925 14:59:06.413623 29785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:06.426447 29785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:06.426759 29785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:06.426910 29785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:06.427142 29785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:06.427467 29785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:06.427520 29785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.427561 29785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:06.427588 29785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:06.433725 29785 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.67:34439
I20260925 14:59:06.433770 29912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.67:34439 every 8 connection(s)
I20260925 14:59:06.434099 29785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:06.438735 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29785
I20260925 14:59:06.438947 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:06.439177 9781 external_mini_cluster.cc:1428] Setting key 5420075b0fa4705f504ad1c69ae8e372
I20260925 14:59:06.439553 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.434822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:42749 (local address 127.9.141.126:44951)
0925 14:59:06.434865 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.434867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.435134 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:59:06.435640 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.435641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.435652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.435721 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:06.436487 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.436965 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.438027 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.438209 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.438262 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.439210 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:06.439211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:06.439212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:06.439217 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:06.439236 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:06.439240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:06.439298 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.439429 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.439456 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:59:06.439780 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.434807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:44951 (local address 127.9.141.67:42749)
0925 14:59:06.434999 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.435039 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.435545 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.435796 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.435799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.435807 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:06.436344 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.436348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.437038 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.437038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.437937 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.437940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.438283 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.438283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.438390 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.439003 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:59:06.439004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:06.439060 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:59:06.439102 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:06.439327 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:06.439335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:06.439338 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.439402 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.439519 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260925 14:59:06.440349 29913 heartbeater.cc:347] Connected to a master server at 127.9.141.126:44951
I20260925 14:59:06.440485 29913 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:06.440627 29913 heartbeater.cc:511] Master 127.9.141.126:44951 requested a full tablet report, sending...
I20260925 14:59:06.440960 29463 ts_manager.cc:194] Registered new tserver with Master: 26ee293068004c2f92772c1c267d7189 (127.9.141.67:34439)
I20260925 14:59:06.441376 29463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.141.67:42749
I20260925 14:59:06.451233 9781 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:06.455670 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.452713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.9.141.126:44951)
0925 14:59:06.452759 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.452761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.452873 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:59:06.452951 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.452952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.452962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.453028 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:06.453216 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.453690 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.454432 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.454625 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.454703 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.455410 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:06.455411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:06.455412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:06.455417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:06.455431 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:06.455435 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:06.455502 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.455564 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.455585 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:06.457348 29463 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:06.458858 29463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:06.465260 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.461349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.9.141.66:39937)
0925 14:59:06.461638 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.461642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.461651 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:06.461702 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.461704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.461720 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.461794 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:06.462014 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.462796 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.464205 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.464852 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.464950 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.465063 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.465132 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.465171 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":85,"threads_started":1}
I20260925 14:59:06.465246 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.461111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:39937 (local address 127.0.0.1:47900)
0925 14:59:06.461352 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.461362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.461415 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.461876 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.461877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.461878 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:06.461934 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.461936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.462878 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.462879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.464115 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.464118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.464955 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.464956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.465029 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.465041 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.465119 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.465172 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260925 14:59:06.465724 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.461438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.9.141.67:34439)
0925 14:59:06.461595 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.461676 (+ 81us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.461973 (+ 297us) server_negotiation.cc:407] Connection header received
0925 14:59:06.462068 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.462070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.462085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.462173 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:06.462283 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.463282 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.464565 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.465263 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.465326 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.465422 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.465601 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.465648 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260925 14:59:06.465688 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.461252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.67:34439 (local address 127.0.0.1:43422)
0925 14:59:06.461883 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.461892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.461947 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.462188 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.462188 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.462190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:06.462221 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.462222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.463359 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.463360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.464467 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.464470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.465453 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.465454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.465504 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.465508 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.465562 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.465604 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":57,"threads_started":1}
I20260925 14:59:06.466151 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.461160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.9.141.65:46423)
0925 14:59:06.461595 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.461600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.461611 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:59:06.461651 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.461653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.461669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.461771 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:06.461928 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.463068 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.464280 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.465043 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.465114 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.465198 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.466027 (+ 829us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.466074 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":103,"threads_started":1}
I20260925 14:59:06.466018 29716 tablet_service.cc:1467] Processing CreateTablet for tablet f6c17d8caaf34e3c8177391c6319b8b8 (DEFAULT_TABLE table=test-table [id=2dc9a766426f44dc8c616a145d28ef7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:06.466419 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c17d8caaf34e3c8177391c6319b8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:06.466413 29847 tablet_service.cc:1467] Processing CreateTablet for tablet f6c17d8caaf34e3c8177391c6319b8b8 (DEFAULT_TABLE table=test-table [id=2dc9a766426f44dc8c616a145d28ef7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:06.466729 29847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c17d8caaf34e3c8177391c6319b8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:06.468528 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.461025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46423 (local address 127.0.0.1:43328)
0925 14:59:06.461239 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.461290 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.461379 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.461775 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.461777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.461779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:06.461844 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.461846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.463159 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.463159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.464202 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.464204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.465236 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.465237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.465602 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.465607 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.465654 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.465701 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260925 14:59:06.469237 29927 tablet_bootstrap.cc:492] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: Bootstrap starting.
I20260925 14:59:06.469218 29585 tablet_service.cc:1467] Processing CreateTablet for tablet f6c17d8caaf34e3c8177391c6319b8b8 (DEFAULT_TABLE table=test-table [id=2dc9a766426f44dc8c616a145d28ef7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:06.469588 29585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c17d8caaf34e3c8177391c6319b8b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:06.469595 29928 tablet_bootstrap.cc:492] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Bootstrap starting.
I20260925 14:59:06.470108 29927 tablet_bootstrap.cc:654] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:06.470283 29928 tablet_bootstrap.cc:654] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:06.470494 29927 log.cc:824] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:06.470661 29928 log.cc:824] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:06.471341 29927 tablet_bootstrap.cc:492] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: No bootstrap required, opened a new log
I20260925 14:59:06.471411 29927 ts_tablet_manager.cc:1397] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:06.471719 29928 tablet_bootstrap.cc:492] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: No bootstrap required, opened a new log
I20260925 14:59:06.471760 29928 ts_tablet_manager.cc:1397] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:06.473127 29931 tablet_bootstrap.cc:492] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a: Bootstrap starting.
I20260925 14:59:06.473778 29931 tablet_bootstrap.cc:654] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:06.473973 29927 raft_consensus.cc:348] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.474102 29931 log.cc:824] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:06.474119 29927 raft_consensus.cc:374] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:06.474159 29927 raft_consensus.cc:729] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f619d68800b949f3a079069fb8e01122, State: Initialized, Role: FOLLOWER
I20260925 14:59:06.474324 29927 consensus_queue.cc:260] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.474349 29928 raft_consensus.cc:348] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.474527 29928 raft_consensus.cc:374] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:06.474568 29928 raft_consensus.cc:729] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ee293068004c2f92772c1c267d7189, State: Initialized, Role: FOLLOWER
I20260925 14:59:06.474632 29782 heartbeater.cc:503] Master 127.9.141.126:44951 was elected leader, sending a full tablet report...
I20260925 14:59:06.474679 29928 consensus_queue.cc:260] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.474594 29927 ts_tablet_manager.cc:1428] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:06.474917 29928 ts_tablet_manager.cc:1428] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:06.475032 29913 heartbeater.cc:503] Master 127.9.141.126:44951 was elected leader, sending a full tablet report...
I20260925 14:59:06.475078 29931 tablet_bootstrap.cc:492] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a: No bootstrap required, opened a new log
I20260925 14:59:06.475143 29931 ts_tablet_manager.cc:1397] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:06.477324 29931 raft_consensus.cc:348] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.477440 29931 raft_consensus.cc:374] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:06.477461 29931 raft_consensus.cc:729] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc8fd68cee1941bd80aaa1fa7604529a, State: Initialized, Role: FOLLOWER
I20260925 14:59:06.477566 29931 consensus_queue.cc:260] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.477809 29931 ts_tablet_manager.cc:1428] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:59:06.477843 29651 heartbeater.cc:503] Master 127.9.141.126:44951 was elected leader, sending a full tablet report...
I20260925 14:59:06.499042 29934 raft_consensus.cc:482] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:06.499167 29934 raft_consensus.cc:504] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.499617 29934 leader_election.cc:288] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc8fd68cee1941bd80aaa1fa7604529a (127.9.141.65:46423), f619d68800b949f3a079069fb8e01122 (127.9.141.66:39937)
I20260925 14:59:06.502460 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.499757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46423 (local address 127.9.141.67:50583)
0925 14:59:06.499818 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.499830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.499875 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.500028 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.500028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.500030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:06.500119 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.500121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.500852 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.500854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.501640 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.501642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.502222 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.502223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.502292 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.502296 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.502349 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.502393 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:59:06.502482 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.499818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:50583 (local address 127.9.141.65:46423)
0925 14:59:06.499864 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.499867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.499877 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:06.499945 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.499946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.499959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.500015 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:06.500219 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.500772 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.501734 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.502134 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.502195 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.502202 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.502370 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.502407 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:59:06.502660 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.499819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.141.67:54677 (local address 127.9.141.66:39937)
0925 14:59:06.499864 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.499867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.500087 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:59:06.500122 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.500123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.500137 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.500234 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:06.500365 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.500913 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.501868 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.502267 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.502340 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.502348 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.502536 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.502583 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:59:06.502882 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.499758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:39937 (local address 127.9.141.67:54677)
0925 14:59:06.500007 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.500018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:06.500079 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:06.500233 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:06.500233 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:06.500235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:06.500301 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.500302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.500992 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.500993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.501777 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:06.501780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.502369 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.502370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:06.502449 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.502453 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:06.502513 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:59:06.502783 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":57,"thread_start_us":90,"threads_started":1}
I20260925 14:59:06.502856 29605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c17d8caaf34e3c8177391c6319b8b8" candidate_uuid: "26ee293068004c2f92772c1c267d7189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" is_pre_election: true
I20260925 14:59:06.503016 29605 raft_consensus.cc:2454] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ee293068004c2f92772c1c267d7189 in term 0.
I20260925 14:59:06.503273 29801 leader_election.cc:302] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ee293068004c2f92772c1c267d7189, dc8fd68cee1941bd80aaa1fa7604529a; no voters:
I20260925 14:59:06.503269 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c17d8caaf34e3c8177391c6319b8b8" candidate_uuid: "26ee293068004c2f92772c1c267d7189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f619d68800b949f3a079069fb8e01122" is_pre_election: true
I20260925 14:59:06.503408 29736 raft_consensus.cc:2454] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ee293068004c2f92772c1c267d7189 in term 0.
I20260925 14:59:06.503424 29934 raft_consensus.cc:2781] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:06.503485 29934 raft_consensus.cc:482] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:06.503509 29934 raft_consensus.cc:3037] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:06.504248 29934 raft_consensus.cc:504] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.504405 29934 leader_election.cc:288] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [CANDIDATE]: Term 1 election: Requested vote from peers dc8fd68cee1941bd80aaa1fa7604529a (127.9.141.65:46423), f619d68800b949f3a079069fb8e01122 (127.9.141.66:39937)
I20260925 14:59:06.504542 29736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c17d8caaf34e3c8177391c6319b8b8" candidate_uuid: "26ee293068004c2f92772c1c267d7189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f619d68800b949f3a079069fb8e01122"
I20260925 14:59:06.504621 29736 raft_consensus.cc:3037] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:06.504624 29605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c17d8caaf34e3c8177391c6319b8b8" candidate_uuid: "26ee293068004c2f92772c1c267d7189" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc8fd68cee1941bd80aaa1fa7604529a"
I20260925 14:59:06.504750 29605 raft_consensus.cc:3037] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:06.505349 29736 raft_consensus.cc:2454] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ee293068004c2f92772c1c267d7189 in term 1.
I20260925 14:59:06.505541 29800 leader_election.cc:302] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ee293068004c2f92772c1c267d7189, f619d68800b949f3a079069fb8e01122; no voters:
I20260925 14:59:06.505586 29605 raft_consensus.cc:2454] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ee293068004c2f92772c1c267d7189 in term 1.
I20260925 14:59:06.505658 29934 raft_consensus.cc:2781] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:06.505841 29934 raft_consensus.cc:686] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 1 LEADER]: Becoming Leader. State: Replica: 26ee293068004c2f92772c1c267d7189, State: Running, Role: LEADER
I20260925 14:59:06.505951 29934 consensus_queue.cc:237] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } }
I20260925 14:59:06.506691 29463 catalog_manager.cc:5495] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ee293068004c2f92772c1c267d7189 (127.9.141.67). New cstate: current_term: 1 leader_uuid: "26ee293068004c2f92772c1c267d7189" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ee293068004c2f92772c1c267d7189" member_type: VOTER last_known_addr { host: "127.9.141.67" port: 34439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:06.522297 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.520001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.9.141.67:34439)
0925 14:59:06.520041 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:06.520043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:06.520144 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:59:06.520215 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:06.520216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:06.520231 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:06.520292 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:06.520472 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.520932 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.521639 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:06.521820 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:06.521870 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:06.522011 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:06.522102 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:06.522159 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:06.522209 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:59:06.522225 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:06.524091 29605 raft_consensus.cc:1261] T f6c17d8caaf34e3c8177391c6319b8b8 P dc8fd68cee1941bd80aaa1fa7604529a [term 1 FOLLOWER]: Refusing update from remote peer 26ee293068004c2f92772c1c267d7189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:06.524091 29736 raft_consensus.cc:1261] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 1 FOLLOWER]: Refusing update from remote peer 26ee293068004c2f92772c1c267d7189: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:06.524513 29934 consensus_queue.cc:1035] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc8fd68cee1941bd80aaa1fa7604529a" member_type: VOTER last_known_addr { host: "127.9.141.65" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:06.524657 29937 consensus_queue.cc:1035] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f619d68800b949f3a079069fb8e01122" member_type: VOTER last_known_addr { host: "127.9.141.66" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:06.527026 29940 mvcc.cc:205] Tried to move back new op lower bound from 7333266827359981568 to 7333266827289059328. Current Snapshot: MvccSnapshot[applied={T|T < 7333266827359981568}]
I20260925 14:59:06.527691 29941 mvcc.cc:205] Tried to move back new op lower bound from 7333266827359981568 to 7333266827289059328. Current Snapshot: MvccSnapshot[applied={T|T < 7333266827359981568}]
I20260925 14:59:06.529158 29463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:06.529248 29463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:06.529979 29463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83ca11965ad74f4e995f85d764d23cea: Sending DeleteTablet for 3 replicas of tablet f6c17d8caaf34e3c8177391c6319b8b8
I20260925 14:59:06.530311 29716 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c17d8caaf34e3c8177391c6319b8b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:06 UTC) from {username='slave'} at 127.0.0.1:47900
I20260925 14:59:06.530318 29585 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c17d8caaf34e3c8177391c6319b8b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:06 UTC) from {username='slave'} at 127.0.0.1:43328
I20260925 14:59:06.530539 29847 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c17d8caaf34e3c8177391c6319b8b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:06 UTC) from {username='slave'} at 127.0.0.1:43422
I20260925 14:59:06.530750 29952 tablet_replica.cc:330] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: stopping tablet replica
I20260925 14:59:06.530884 29952 raft_consensus.cc:2229] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:06.530998 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29523
I20260925 14:59:06.531512 29952 raft_consensus.cc:2258] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:06.536835 29952 ts_tablet_manager.cc:1905] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:06.537127 29950 tablet_replica.cc:330] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122: stopping tablet replica
W20260925 14:59:06.537230 29448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.141.65:46423 (error 108)
I20260925 14:59:06.537355 29950 raft_consensus.cc:2229] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:06.537544 29950 raft_consensus.cc:2258] T f6c17d8caaf34e3c8177391c6319b8b8 P f619d68800b949f3a079069fb8e01122 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:06.537773 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29654
I20260925 14:59:06.538908 29952 ts_tablet_manager.cc:1918] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:06.538986 29952 log.cc:1196] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Deleting WAL directory at /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348105363960-9781-0/minicluster-data/ts-2/wal/wals/f6c17d8caaf34e3c8177391c6319b8b8
I20260925 14:59:06.539275 29952 ts_tablet_manager.cc:1939] T f6c17d8caaf34e3c8177391c6319b8b8 P 26ee293068004c2f92772c1c267d7189: Deleting consensus metadata
I20260925 14:59:06.540139 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.539964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.65:46423 (local address 127.0.0.1:43336)
0925 14:59:06.540050 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.540095 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.65:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 14:59:06.540269 29448 catalog_manager.cc:4584] TS dc8fd68cee1941bd80aaa1fa7604529a (127.9.141.65:46423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6c17d8caaf34e3c8177391c6319b8b8: Network error: Client connection negotiation failed: client connection to 127.9.141.65:46423: connect: Connection refused (error 111)
I20260925 14:59:06.540369 29448 catalog_manager.cc:4841] TS 26ee293068004c2f92772c1c267d7189 (127.9.141.67:34439): tablet f6c17d8caaf34e3c8177391c6319b8b8 (table test-table [id=2dc9a766426f44dc8c616a145d28ef7b]) successfully deleted
I20260925 14:59:06.543318 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29785
I20260925 14:59:06.543372 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:06.543247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.66:39937 (local address 127.0.0.1:47908)
0925 14:59:06.543304 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:06.543336 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.66:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:59:06.548028 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29432
2026-09-25T14:59:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (737 ms)
[----------] 6 tests from EncryptionPolicyTest (8610 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:06 dist-test-slave-1sv7 krb5kdc[29956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:06 dist-test-slave-1sv7 krb5kdc[29956](info): set up 2 sockets
Sep 25 14:59:06 dist-test-slave-1sv7 krb5kdc[29956](info): commencing operation
krb5kdc: starting...
W20260925 14:59:07.608345 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:07 dist-test-slave-1sv7 krb5kdc[29956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:07.736790 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:07.817649 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:07.817900 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:07.817938 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:07.819085 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:07.819159 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:07.819182 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:07.819204 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:07.819232 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:07.820735 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:07.821029 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:07.821261 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:07.826190 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:07.826198 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:07.826205 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:07.826406 29969 server_base.cc:1034] running on GCE node
I20260925 14:59:07.826714 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:07.826988 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:07.828167 29969 hybrid_clock.cc:648] HybridClock initialized: now 1790348347828147 us; error 34 us; skew 500 ppm
Sep 25 14:59:07 dist-test-slave-1sv7 krb5kdc[29956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:07.830814 29969 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:07.831351 29969 webserver.cc:460] Webserver started at http://127.0.0.1:34291/ using document root <none> and password file <none>
I20260925 14:59:07.831600 29969 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:07.831676 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:07.831835 29969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:07.833089 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "abe96b2d9e8249a19138cf10632b238a"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-1sv7"
server_key: "c26a32179364a4bf97a76529c06cab25"
server_key_iv: "c2577fc0fe9ef37897bb84aecf02b6f2"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.833478 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "abe96b2d9e8249a19138cf10632b238a"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-1sv7"
server_key: "c26a32179364a4bf97a76529c06cab25"
server_key_iv: "c2577fc0fe9ef37897bb84aecf02b6f2"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.834880 29969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:07.835727 29984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:07.835983 29969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:07.836102 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "abe96b2d9e8249a19138cf10632b238a"
format_stamp: "Formatted at 2026-09-25 14:59:07 on dist-test-slave-1sv7"
server_key: "c26a32179364a4bf97a76529c06cab25"
server_key_iv: "c2577fc0fe9ef37897bb84aecf02b6f2"
server_key_version: "encryptionkey@0"
I20260925 14:59:07.836185 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:07.850503 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:07.852993 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:07.853389 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:07.858415 29969 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46509
I20260925 14:59:07.858451 30036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46509 every 8 connection(s)
I20260925 14:59:07.858879 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:07.859836 30037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:07.861802 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 29969
I20260925 14:59:07.861929 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:07.862115 9781 external_mini_cluster.cc:1428] Setting key e840183db94e8e95bd8d4f03ea46810f
I20260925 14:59:07.862869 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a: Bootstrap starting.
I20260925 14:59:07.863528 30037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:07.863886 30037 log.cc:824] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:07.864758 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a: No bootstrap required, opened a new log
I20260925 14:59:07.866940 30037 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe96b2d9e8249a19138cf10632b238a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260925 14:59:07.867079 30037 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:07.867102 30037 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe96b2d9e8249a19138cf10632b238a, State: Initialized, Role: FOLLOWER
I20260925 14:59:07.867306 30037 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe96b2d9e8249a19138cf10632b238a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260925 14:59:07.867377 30037 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:07.867422 30037 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:07.867471 30037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:07.868353 30037 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe96b2d9e8249a19138cf10632b238a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
I20260925 14:59:07.868526 30037 leader_election.cc:302] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe96b2d9e8249a19138cf10632b238a; no voters:
I20260925 14:59:07.868710 30037 leader_election.cc:288] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:07.868759 30042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:07.868984 30042 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [term 1 LEADER]: Becoming Leader. State: Replica: abe96b2d9e8249a19138cf10632b238a, State: Running, Role: LEADER
I20260925 14:59:07.868985 30037 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:07.869117 30042 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe96b2d9e8249a19138cf10632b238a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } }
Sep 25 14:59:07 dist-test-slave-1sv7 krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:59:07.869529 30044 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe96b2d9e8249a19138cf10632b238a. Latest consensus state: current_term: 1 leader_uuid: "abe96b2d9e8249a19138cf10632b238a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe96b2d9e8249a19138cf10632b238a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } } }
I20260925 14:59:07.869616 30044 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:07.869869 30049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:07.870714 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe96b2d9e8249a19138cf10632b238a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe96b2d9e8249a19138cf10632b238a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46509 } } }
I20260925 14:59:07.870793 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:07.871546 30049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:07.875013 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.862712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59029 (local address 127.0.0.1:46509)
0925 14:59:07.862900 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:07.862907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:07.862957 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:59:07.863407 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:07.863413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:07.863470 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:07.863630 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:07.864461 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.865502 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.866624 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.867063 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.867205 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.870155 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:07.870160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:07.870161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:07.870174 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:07.872904 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.873437 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.873440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.873440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.873528 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.873844 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.873845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.873845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.874330 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:07.874534 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:07.874712 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:59:07.874819 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260925 14:59:07.876071 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:07.880954 30049 catalog_manager.cc:1347] Generated new cluster ID: 4b4af4f5aeb9418ca8646602d9a5f535
I20260925 14:59:07.881024 30049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:07.885296 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:07.877422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47801 (local address 127.0.0.1:46509)
0925 14:59:07.877458 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:07.877460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:07.877627 (+ 167us) server_negotiation.cc:407] Connection header received
0925 14:59:07.877702 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:07.877703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:07.877710 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:07.877758 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:07.878472 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.879282 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.880477 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:07.880790 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:07.880866 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:07.882406 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:07.882407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:07.882408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:07.882414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:07.884020 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.884476 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.884478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.884478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.884529 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:07.884736 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:07.884737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:07.884737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:07.884850 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:07.884918 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:07.885144 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 14:59:07.885206 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:59:07.892527 30049 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:07.893064 30049 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:07.900130 30049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe96b2d9e8249a19138cf10632b238a: Generated new TSK 0
I20260925 14:59:07.900323 30049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:07.904695 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 29969
W20260925 14:59:07.908993 29954 connection.cc:535] client connection to 127.0.0.1:46509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46509): Connection reset by peer (error 104)
2026-09-25T14:59:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:07 dist-test-slave-1sv7 krb5kdc[30068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:07 dist-test-slave-1sv7 krb5kdc[30068](info): set up 2 sockets
Sep 25 14:59:07 dist-test-slave-1sv7 krb5kdc[30068](info): commencing operation
krb5kdc: starting...
W20260925 14:59:08.964524 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:09 dist-test-slave-1sv7 krb5kdc[30068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:09.092185 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:09.174100 30081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:09.174316 30081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:09.174341 30081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:09.175606 30081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:09.175653 30081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:09.175668 30081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:09.175680 30081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:09.175693 30081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:09.177237 30081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:09.177492 30081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:09.177728 30081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:09.183007 30087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:09.183004 30086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:09.183000 30089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:09.183116 30081 server_base.cc:1034] running on GCE node
I20260925 14:59:09.183490 30081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:09.183790 30081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:09.185053 30081 hybrid_clock.cc:648] HybridClock initialized: now 1790348349184963 us; error 38 us; skew 500 ppm
Sep 25 14:59:09 dist-test-slave-1sv7 krb5kdc[30068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:09.187707 30081 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:09.188091 30081 webserver.cc:460] Webserver started at http://127.0.0.1:38729/ using document root <none> and password file <none>
I20260925 14:59:09.188319 30081 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:09.188364 30081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:09.188577 30081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:09.189634 30081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "56158dd0969c470ea79e81c70eaf09e4"
format_stamp: "Formatted at 2026-09-25 14:59:09 on dist-test-slave-1sv7"
server_key: "84d6c904ed76d8a9ed0c0f6fe5bcf459"
server_key_iv: "393a0e241149c4ccfa66b1a1d794250f"
server_key_version: "encryptionkey@0"
I20260925 14:59:09.189998 30081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "56158dd0969c470ea79e81c70eaf09e4"
format_stamp: "Formatted at 2026-09-25 14:59:09 on dist-test-slave-1sv7"
server_key: "84d6c904ed76d8a9ed0c0f6fe5bcf459"
server_key_iv: "393a0e241149c4ccfa66b1a1d794250f"
server_key_version: "encryptionkey@0"
I20260925 14:59:09.191335 30081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:59:09.192165 30096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:09.192423 30081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:09.192536 30081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "56158dd0969c470ea79e81c70eaf09e4"
format_stamp: "Formatted at 2026-09-25 14:59:09 on dist-test-slave-1sv7"
server_key: "84d6c904ed76d8a9ed0c0f6fe5bcf459"
server_key_iv: "393a0e241149c4ccfa66b1a1d794250f"
server_key_version: "encryptionkey@0"
I20260925 14:59:09.192628 30081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:09.209281 30081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:09.211261 30081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:09.211413 30081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:09.215499 30081 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39367
I20260925 14:59:09.215577 30148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39367 every 8 connection(s)
I20260925 14:59:09.215916 30081 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:09.216948 30149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:09.217182 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30081
I20260925 14:59:09.217301 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:09.217526 9781 external_mini_cluster.cc:1428] Setting key aefce32ec75cf283c7262545cf96de73
I20260925 14:59:09.220404 30149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4: Bootstrap starting.
I20260925 14:59:09.221114 30149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:09.221421 30149 log.cc:824] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:09.222194 30149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4: No bootstrap required, opened a new log
I20260925 14:59:09.223170 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.218060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51291 (local address 127.0.0.1:39367)
0925 14:59:09.218269 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.218276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.218300 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:59:09.218716 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.218723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.218766 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.218876 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:09.219129 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.219999 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.221086 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.221591 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.221731 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.222508 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:09.222512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:09.222512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:09.222696 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:09.222731 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:09.222745 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:09.222851 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.222935 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.222963 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
I20260925 14:59:09.224299 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:09.224539 30149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56158dd0969c470ea79e81c70eaf09e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260925 14:59:09.224694 30149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:09.224742 30149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56158dd0969c470ea79e81c70eaf09e4, State: Initialized, Role: FOLLOWER
I20260925 14:59:09.224852 30149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56158dd0969c470ea79e81c70eaf09e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260925 14:59:09.224920 30149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:09.224942 30149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:09.225016 30149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:09.225735 30149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56158dd0969c470ea79e81c70eaf09e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260925 14:59:09.225900 30149 leader_election.cc:302] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56158dd0969c470ea79e81c70eaf09e4; no voters:
I20260925 14:59:09.226092 30149 leader_election.cc:288] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:09.226125 30159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:09.226238 30159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [term 1 LEADER]: Becoming Leader. State: Replica: 56158dd0969c470ea79e81c70eaf09e4, State: Running, Role: LEADER
I20260925 14:59:09.226361 30149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:09.226375 30159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56158dd0969c470ea79e81c70eaf09e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } }
I20260925 14:59:09.226719 30159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56158dd0969c470ea79e81c70eaf09e4. Latest consensus state: current_term: 1 leader_uuid: "56158dd0969c470ea79e81c70eaf09e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56158dd0969c470ea79e81c70eaf09e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260925 14:59:09.226807 30159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:09.226735 30161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56158dd0969c470ea79e81c70eaf09e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56158dd0969c470ea79e81c70eaf09e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39367 } } }
I20260925 14:59:09.226943 30161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:09.227031 30165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:09.227599 30165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:09.229471 30165 catalog_manager.cc:1347] Generated new cluster ID: 07905c6e706b4ec29254201f9214b3cb
I20260925 14:59:09.229527 30165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:09.229601 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:09.225832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39511 (local address 127.0.0.1:39367)
0925 14:59:09.225873 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:09.225875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:09.225939 (+ 64us) server_negotiation.cc:407] Connection header received
0925 14:59:09.225995 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:09.225996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:09.226006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:09.226072 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:09.226297 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.226997 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.228121 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:09.228436 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:09.228477 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:09.229248 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:09.229250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:09.229250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:09.229255 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:09.229270 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:09.229275 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:09.229354 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:09.229488 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:59:09.229500 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:59:09.248606 30165 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:09.249192 30165 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:09.253353 30165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56158dd0969c470ea79e81c70eaf09e4: Generated new TSK 0
I20260925 14:59:09.253587 30165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:09.259577 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30081
W20260925 14:59:09.264029 30066 connection.cc:535] client connection to 127.0.0.1:39367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39367): Connection reset by peer (error 104)
2026-09-25T14:59:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:09 dist-test-slave-1sv7 krb5kdc[30180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:09 dist-test-slave-1sv7 krb5kdc[30180](info): set up 2 sockets
Sep 25 14:59:09 dist-test-slave-1sv7 krb5kdc[30180](info): commencing operation
krb5kdc: starting...
W20260925 14:59:10.312772 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:10 dist-test-slave-1sv7 krb5kdc[30180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:10.441542 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:10.522221 30193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:10.522485 30193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:10.522533 30193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:10.523746 30193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:10.523829 30193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:10.523854 30193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:10.523880 30193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:10.523911 30193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:10.525426 30193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:10.525715 30193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:10.525964 30193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:10.530977 30201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:10.531085 30193 server_base.cc:1034] running on GCE node
W20260925 14:59:10.531046 30198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:10.530967 30199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:10.531605 30193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:10.531937 30193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:10.533106 30193 hybrid_clock.cc:648] HybridClock initialized: now 1790348350533134 us; error 72 us; skew 500 ppm
Sep 25 14:59:10 dist-test-slave-1sv7 krb5kdc[30180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:10.544354 30193 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:10.545029 30193 webserver.cc:460] Webserver started at http://127.0.0.1:41533/ using document root <none> and password file <none>
I20260925 14:59:10.545331 30193 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:10.545385 30193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:10.545542 30193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:10.546741 30193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58"
format_stamp: "Formatted at 2026-09-25 14:59:10 on dist-test-slave-1sv7"
server_key: "d34f541d6b083935537bccf8cc9b1b0c"
server_key_iv: "000175072376d73a0733fc2d7d26e277"
server_key_version: "encryptionkey@0"
I20260925 14:59:10.547168 30193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58"
format_stamp: "Formatted at 2026-09-25 14:59:10 on dist-test-slave-1sv7"
server_key: "d34f541d6b083935537bccf8cc9b1b0c"
server_key_iv: "000175072376d73a0733fc2d7d26e277"
server_key_version: "encryptionkey@0"
I20260925 14:59:10.548753 30193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:10.549724 30208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:10.549944 30193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:10.550093 30193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58"
format_stamp: "Formatted at 2026-09-25 14:59:10 on dist-test-slave-1sv7"
server_key: "d34f541d6b083935537bccf8cc9b1b0c"
server_key_iv: "000175072376d73a0733fc2d7d26e277"
server_key_version: "encryptionkey@0"
I20260925 14:59:10.550156 30193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:10.553596 30193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:10.555964 30193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:10.556182 30193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:10.559898 30193 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40647
I20260925 14:59:10.559952 30260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40647 every 8 connection(s)
I20260925 14:59:10.560325 30193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:10.561489 30261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:10.563953 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58: Bootstrap starting.
I20260925 14:59:10.564687 30261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:10.565011 30261 log.cc:824] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:10.565834 30261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58: No bootstrap required, opened a new log
I20260925 14:59:10.566502 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30193
I20260925 14:59:10.566627 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:10.566848 9781 external_mini_cluster.cc:1428] Setting key f9657e374122131f7951e6d2e6b13126
I20260925 14:59:10.568778 30261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260925 14:59:10.568981 30261 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:10.569051 30261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4ca7c3698340a5ae52f8ba0f79aa58, State: Initialized, Role: FOLLOWER
I20260925 14:59:10.569216 30261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260925 14:59:10.569332 30261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:10.569370 30261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:10.569427 30261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:10.570011 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.567440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55171 (local address 127.0.0.1:40647)
0925 14:59:10.567657 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.567666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.567705 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:59:10.568702 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.568710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.568738 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.568896 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:10.569592 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:10.569595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:10.569596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:10.569600 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:10.569647 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:10.569658 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:10.569741 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.569810 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.569850 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260925 14:59:10.570184 30261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260925 14:59:10.570335 30261 leader_election.cc:302] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4ca7c3698340a5ae52f8ba0f79aa58; no voters:
I20260925 14:59:10.570634 30261 leader_election.cc:288] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:10.570818 30267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:10.570909 30261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:10.571049 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:10.570982 30267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [term 1 LEADER]: Becoming Leader. State: Replica: 9b4ca7c3698340a5ae52f8ba0f79aa58, State: Running, Role: LEADER
I20260925 14:59:10.572625 30267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } }
I20260925 14:59:10.573203 30268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } } }
I20260925 14:59:10.573292 30266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4ca7c3698340a5ae52f8ba0f79aa58. Latest consensus state: current_term: 1 leader_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4ca7c3698340a5ae52f8ba0f79aa58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40647 } } }
I20260925 14:59:10.573479 30268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:10.573539 30266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:10.573681 30265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:10.572477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47161 (local address 127.0.0.1:40647)
0925 14:59:10.572516 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:10.572518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:10.572640 (+ 122us) server_negotiation.cc:407] Connection header received
0925 14:59:10.572696 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:10.572697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:10.572701 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:10.572759 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:10.573411 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:10.573412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:10.573412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:10.573417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:10.573432 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:10.573436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:10.573470 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:10.573599 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:59:10.573614 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:59:10.574209 30285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:10.574270 30285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:10.574432 30287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:10.574627 30287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:10.576306 30287 catalog_manager.cc:1347] Generated new cluster ID: cfc6b0f865c44a918897a1fd269e96c4
I20260925 14:59:10.576366 30287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:10.596194 30287 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:10.596426 30287 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:10.606473 30287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b4ca7c3698340a5ae52f8ba0f79aa58: Generated new TSK 0
I20260925 14:59:10.606698 30287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:10.614143 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30193
2026-09-25T14:59:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:10 dist-test-slave-1sv7 krb5kdc[30293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:10 dist-test-slave-1sv7 krb5kdc[30293](info): set up 2 sockets
Sep 25 14:59:10 dist-test-slave-1sv7 krb5kdc[30293](info): commencing operation
krb5kdc: starting...
W20260925 14:59:11.656478 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:11 dist-test-slave-1sv7 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:11.784665 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:11.866669 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:11.866907 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:11.866933 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:11.868084 30306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:11.868122 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:11.868136 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:11.868149 30306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:11.868161 30306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:11.869686 30306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:11.869904 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:11.870081 30306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:11.875223 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:11.875353 30306 server_base.cc:1034] running on GCE node
W20260925 14:59:11.875303 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:11.875240 30314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:11.875697 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:11.876015 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:11.877214 30306 hybrid_clock.cc:648] HybridClock initialized: now 1790348351877184 us; error 57 us; skew 500 ppm
Sep 25 14:59:11 dist-test-slave-1sv7 krb5kdc[30293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:11.880144 30306 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:11.880751 30306 webserver.cc:460] Webserver started at http://127.0.0.1:34187/ using document root <none> and password file <none>
I20260925 14:59:11.880992 30306 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:11.881039 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:11.881131 30306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:11.882200 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "f7f6afe9e60b4be4a07901ad03015cf1"
format_stamp: "Formatted at 2026-09-25 14:59:11 on dist-test-slave-1sv7"
server_key: "fe9f0a0182be1a1449da07b682175fcb"
server_key_iv: "514e3c374241debe47253e9cfba70e74"
server_key_version: "encryptionkey@0"
I20260925 14:59:11.882527 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "f7f6afe9e60b4be4a07901ad03015cf1"
format_stamp: "Formatted at 2026-09-25 14:59:11 on dist-test-slave-1sv7"
server_key: "fe9f0a0182be1a1449da07b682175fcb"
server_key_iv: "514e3c374241debe47253e9cfba70e74"
server_key_version: "encryptionkey@0"
I20260925 14:59:11.883863 30306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:11.884860 30321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:11.885118 30306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:11.885218 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "f7f6afe9e60b4be4a07901ad03015cf1"
format_stamp: "Formatted at 2026-09-25 14:59:11 on dist-test-slave-1sv7"
server_key: "fe9f0a0182be1a1449da07b682175fcb"
server_key_iv: "514e3c374241debe47253e9cfba70e74"
server_key_version: "encryptionkey@0"
I20260925 14:59:11.885278 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:11.910441 30306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:11.912775 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:11.912895 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:11.916865 30306 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36643
I20260925 14:59:11.916971 30373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36643 every 8 connection(s)
I20260925 14:59:11.917299 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:11.918495 30374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:11.919888 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30306
I20260925 14:59:11.919971 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:11.920168 9781 external_mini_cluster.cc:1428] Setting key d4b5202ba894303e63f02d9ca83d75e1
I20260925 14:59:11.921346 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1: Bootstrap starting.
I20260925 14:59:11.922093 30374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:11.922451 30374 log.cc:824] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:11.923190 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1: No bootstrap required, opened a new log
I20260925 14:59:11.925599 30374 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260925 14:59:11.925793 30374 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:11.925836 30374 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f6afe9e60b4be4a07901ad03015cf1, State: Initialized, Role: FOLLOWER
I20260925 14:59:11.925978 30374 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260925 14:59:11.926070 30374 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:11.926106 30374 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:11.926159 30374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:11.926934 30374 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260925 14:59:11.927080 30374 leader_election.cc:302] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f6afe9e60b4be4a07901ad03015cf1; no voters:
I20260925 14:59:11.927425 30374 leader_election.cc:288] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:11.927448 30379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:11.927579 30379 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [term 1 LEADER]: Becoming Leader. State: Replica: f7f6afe9e60b4be4a07901ad03015cf1, State: Running, Role: LEADER
Sep 25 14:59:11 dist-test-slave-1sv7 krb5kdc[30293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:59:11.927716 30374 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:11.927732 30379 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } }
I20260925 14:59:11.928134 30379 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f6afe9e60b4be4a07901ad03015cf1. Latest consensus state: current_term: 1 leader_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } } }
I20260925 14:59:11.928397 30379 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:11.928293 30380 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6afe9e60b4be4a07901ad03015cf1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36643 } } }
I20260925 14:59:11.928676 30380 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:11.929358 30388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:11.930053 30388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:11.931602 30388 catalog_manager.cc:1347] Generated new cluster ID: 382f7b4c532044c1877a53dbc68b3de2
I20260925 14:59:11.931649 30388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:11.931924 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.920793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40501 (local address 127.0.0.1:36643)
0925 14:59:11.921007 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:11.921070 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:11.921097 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:59:11.921532 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:11.921541 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:11.921597 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:11.921713 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:11.922612 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.923601 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.924699 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.925098 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.925181 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.928169 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:11.928173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:11.928173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:11.928185 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:11.930204 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.930628 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.930639 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.930639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.930740 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.930974 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.930975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.930976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.931305 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:11.931460 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.931620 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.931754 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":119,"threads_started":1}
I20260925 14:59:11.933585 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:11.942214 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:11.935168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38331 (local address 127.0.0.1:36643)
0925 14:59:11.935208 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:11.935210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:11.935367 (+ 157us) server_negotiation.cc:407] Connection header received
0925 14:59:11.935411 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:11.935412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:11.935421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:11.935476 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:11.936023 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.936623 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.937387 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:11.937634 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:11.937726 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:11.939231 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:11.939232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:11.939233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:11.939240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:11.940943 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.941352 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.941353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.941353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.941415 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:11.941642 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:11.941643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:11.941643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:11.941770 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:11.941889 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:11.942014 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 14:59:11.942100 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:11.946717 30388 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:11.947330 30388 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:11.959851 30388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f6afe9e60b4be4a07901ad03015cf1: Generated new TSK 0
I20260925 14:59:11.960101 30388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:11.967551 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30306
2026-09-25T14:59:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:12 dist-test-slave-1sv7 krb5kdc[30405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:12 dist-test-slave-1sv7 krb5kdc[30405](info): set up 2 sockets
Sep 25 14:59:12 dist-test-slave-1sv7 krb5kdc[30405](info): commencing operation
krb5kdc: starting...
W20260925 14:59:13.012604 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:13 dist-test-slave-1sv7 krb5kdc[30405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:13.143008 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:13.226060 30418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:13.226271 30418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:13.226294 30418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:13.227553 30418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:13.227596 30418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:13.227610 30418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:13.227623 30418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:13.227635 30418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:13.229152 30418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:13.229418 30418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:13.229665 30418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:13.234978 30423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:13.234978 30424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:13.235098 30426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:13.235258 30418 server_base.cc:1034] running on GCE node
I20260925 14:59:13.235662 30418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:13.235961 30418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:13.237169 30418 hybrid_clock.cc:648] HybridClock initialized: now 1790348353237154 us; error 34 us; skew 500 ppm
Sep 25 14:59:13 dist-test-slave-1sv7 krb5kdc[30405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:13.239702 30418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:13.240221 30418 webserver.cc:460] Webserver started at http://127.0.0.1:38953/ using document root <none> and password file <none>
I20260925 14:59:13.240535 30418 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:13.240605 30418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:13.240743 30418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:13.241853 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "b465de379dd64ad3a6db313770523321"
format_stamp: "Formatted at 2026-09-25 14:59:13 on dist-test-slave-1sv7"
server_key: "21c68b6d8244031bfa05a4b58e1f7fbc"
server_key_iv: "1dfe5ec7ac1ac59ad4327e98e8ae17fd"
server_key_version: "encryptionkey@0"
I20260925 14:59:13.242225 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "b465de379dd64ad3a6db313770523321"
format_stamp: "Formatted at 2026-09-25 14:59:13 on dist-test-slave-1sv7"
server_key: "21c68b6d8244031bfa05a4b58e1f7fbc"
server_key_iv: "1dfe5ec7ac1ac59ad4327e98e8ae17fd"
server_key_version: "encryptionkey@0"
I20260925 14:59:13.243556 30418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:13.244372 30433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:13.244637 30418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:13.244737 30418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "b465de379dd64ad3a6db313770523321"
format_stamp: "Formatted at 2026-09-25 14:59:13 on dist-test-slave-1sv7"
server_key: "21c68b6d8244031bfa05a4b58e1f7fbc"
server_key_iv: "1dfe5ec7ac1ac59ad4327e98e8ae17fd"
server_key_version: "encryptionkey@0"
I20260925 14:59:13.244824 30418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:13.280778 30418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:13.282963 30418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:13.283128 30418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:13.287891 30418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36023
I20260925 14:59:13.287918 30485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36023 every 8 connection(s)
I20260925 14:59:13.288334 30418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:13.289461 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:13.291759 30486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321: Bootstrap starting.
I20260925 14:59:13.292464 30486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:13.292799 30486 log.cc:824] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:13.293675 30486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321: No bootstrap required, opened a new log
I20260925 14:59:13.295219 30486 raft_consensus.cc:348] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b465de379dd64ad3a6db313770523321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260925 14:59:13.295351 30486 raft_consensus.cc:374] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:13.295373 30486 raft_consensus.cc:729] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b465de379dd64ad3a6db313770523321, State: Initialized, Role: FOLLOWER
I20260925 14:59:13.295477 30486 consensus_queue.cc:260] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b465de379dd64ad3a6db313770523321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260925 14:59:13.295526 30486 raft_consensus.cc:388] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:13.295550 30486 raft_consensus.cc:482] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:13.295581 30486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:13.296067 30486 raft_consensus.cc:504] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b465de379dd64ad3a6db313770523321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260925 14:59:13.296151 30486 leader_election.cc:302] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b465de379dd64ad3a6db313770523321; no voters:
I20260925 14:59:13.296316 30486 leader_election.cc:288] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:13.296345 30489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:13.296499 30489 raft_consensus.cc:686] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [term 1 LEADER]: Becoming Leader. State: Replica: b465de379dd64ad3a6db313770523321, State: Running, Role: LEADER
I20260925 14:59:13.296624 30486 sys_catalog.cc:564] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:13.296650 30489 consensus_queue.cc:237] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b465de379dd64ad3a6db313770523321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } }
I20260925 14:59:13.297066 30491 sys_catalog.cc:455] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b465de379dd64ad3a6db313770523321. Latest consensus state: current_term: 1 leader_uuid: "b465de379dd64ad3a6db313770523321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b465de379dd64ad3a6db313770523321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } } }
I20260925 14:59:13.297169 30491 sys_catalog.cc:458] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:13.297187 30490 sys_catalog.cc:455] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b465de379dd64ad3a6db313770523321" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b465de379dd64ad3a6db313770523321" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36023 } } }
I20260925 14:59:13.297287 30490 sys_catalog.cc:458] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:13.297510 30497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:13.298058 30497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:13.298370 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30418
I20260925 14:59:13.298477 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:13.298671 9781 external_mini_cluster.cc:1428] Setting key 0beca147a86e2931d02f8e9fa4355596
I20260925 14:59:13.299710 30497 catalog_manager.cc:1347] Generated new cluster ID: 7f585b32627847178f70b7e57f0f944b
I20260925 14:59:13.299765 30497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:13.305274 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:13.299281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47375 (local address 127.0.0.1:36023)
0925 14:59:13.299445 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:13.299454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:13.299485 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:59:13.299600 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:13.299607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:13.299664 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:13.299800 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:13.300141 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:13.301613 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:13.302697 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:13.303344 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:13.303463 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:13.304220 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:13.304224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:13.304224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:13.304487 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:13.304529 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:13.304547 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:13.304617 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:13.304687 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:59:13.304852 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260925 14:59:13.306126 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:13.309994 30497 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:13.310662 30497 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:13.311187 30509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:13.307527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56709 (local address 127.0.0.1:36023)
0925 14:59:13.307584 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:13.307586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:13.307778 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:59:13.307832 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:13.307833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:13.307844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:13.307894 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:13.308053 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:13.308791 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:13.309824 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:13.310147 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:13.310224 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:13.310936 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:13.310937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:13.310937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:13.310942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:13.310955 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:13.310959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:13.311013 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:13.311072 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:59:13.311110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:59:13.315435 30497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b465de379dd64ad3a6db313770523321: Generated new TSK 0
I20260925 14:59:13.315640 30497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:13.318081 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30418
2026-09-25T14:59:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:13 dist-test-slave-1sv7 krb5kdc[30517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:13 dist-test-slave-1sv7 krb5kdc[30517](info): set up 2 sockets
Sep 25 14:59:13 dist-test-slave-1sv7 krb5kdc[30517](info): commencing operation
krb5kdc: starting...
W20260925 14:59:14.376458 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:14 dist-test-slave-1sv7 krb5kdc[30517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:14.496786 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:14.579212 30530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:14.579417 30530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:14.579439 30530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:14.580534 30530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:14.580569 30530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:14.580585 30530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:14.580597 30530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:14.580610 30530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:14.582041 30530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:14.582226 30530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:14.582389 30530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:14.587491 30535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:14.587499 30536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:14.587630 30530 server_base.cc:1034] running on GCE node
W20260925 14:59:14.587653 30538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:14.588248 30530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:14.588600 30530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:14.589787 30530 hybrid_clock.cc:648] HybridClock initialized: now 1790348354589771 us; error 33 us; skew 500 ppm
Sep 25 14:59:14 dist-test-slave-1sv7 krb5kdc[30517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:14.592689 30530 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:14.593206 30530 webserver.cc:460] Webserver started at http://127.0.0.1:40921/ using document root <none> and password file <none>
I20260925 14:59:14.593451 30530 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:14.593519 30530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:14.593655 30530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:14.594730 30530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "eb3f6719b5fd4589889be104691c38ec"
format_stamp: "Formatted at 2026-09-25 14:59:14 on dist-test-slave-1sv7"
server_key: "e899b0e7fc1438a6cda3f0579e452474"
server_key_iv: "6179df74efdcd8a1f3c91944f587d554"
server_key_version: "encryptionkey@0"
I20260925 14:59:14.595108 30530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "eb3f6719b5fd4589889be104691c38ec"
format_stamp: "Formatted at 2026-09-25 14:59:14 on dist-test-slave-1sv7"
server_key: "e899b0e7fc1438a6cda3f0579e452474"
server_key_iv: "6179df74efdcd8a1f3c91944f587d554"
server_key_version: "encryptionkey@0"
I20260925 14:59:14.596940 30530 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:14.597781 30545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:14.598074 30530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:14.598184 30530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "eb3f6719b5fd4589889be104691c38ec"
format_stamp: "Formatted at 2026-09-25 14:59:14 on dist-test-slave-1sv7"
server_key: "e899b0e7fc1438a6cda3f0579e452474"
server_key_iv: "6179df74efdcd8a1f3c91944f587d554"
server_key_version: "encryptionkey@0"
I20260925 14:59:14.598273 30530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:14.601768 30530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:14.604010 30530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:14.604225 30530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:14.608667 30530 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42655
I20260925 14:59:14.608695 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42655 every 8 connection(s)
I20260925 14:59:14.609048 30530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:14.610136 30598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:14.611971 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30530
I20260925 14:59:14.612094 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:14.612295 9781 external_mini_cluster.cc:1428] Setting key c2b39acdd63e128ce789da7db46f0e5e
I20260925 14:59:14.612879 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec: Bootstrap starting.
I20260925 14:59:14.613606 30598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:14.613945 30598 log.cc:824] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:14.614756 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec: No bootstrap required, opened a new log
I20260925 14:59:14.615260 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:14.612869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46549 (local address 127.0.0.1:42655)
0925 14:59:14.613066 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:14.613075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:14.613107 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:59:14.613767 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:14.613778 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:14.613819 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:14.613977 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:14.614712 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:14.614715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:14.614716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:14.614720 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:14.614760 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:14.614770 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:14.614893 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:14.615045 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 14:59:14.615091 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260925 14:59:14.616245 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:14.617025 30598 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3f6719b5fd4589889be104691c38ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260925 14:59:14.617228 30598 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:14.617281 30598 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3f6719b5fd4589889be104691c38ec, State: Initialized, Role: FOLLOWER
I20260925 14:59:14.617408 30598 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3f6719b5fd4589889be104691c38ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260925 14:59:14.617507 30598 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:14.617551 30598 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:14.617599 30598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:14.618114 30598 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3f6719b5fd4589889be104691c38ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260925 14:59:14.618232 30598 leader_election.cc:302] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb3f6719b5fd4589889be104691c38ec; no voters:
I20260925 14:59:14.618438 30598 leader_election.cc:288] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:14.618461 30608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:14.618639 30608 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [term 1 LEADER]: Becoming Leader. State: Replica: eb3f6719b5fd4589889be104691c38ec, State: Running, Role: LEADER
I20260925 14:59:14.618716 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:14.617733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60781 (local address 127.0.0.1:42655)
0925 14:59:14.617766 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:14.617768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:14.617776 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:14.617820 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:14.617820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:14.617825 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:14.617888 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:14.618491 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:14.618492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:14.618493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:14.618497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:14.618511 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:14.618515 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:14.618571 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:14.618651 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:59:14.618665 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:14.618739 30598 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:14.618762 30608 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3f6719b5fd4589889be104691c38ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } }
I20260925 14:59:14.619071 30608 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb3f6719b5fd4589889be104691c38ec. Latest consensus state: current_term: 1 leader_uuid: "eb3f6719b5fd4589889be104691c38ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3f6719b5fd4589889be104691c38ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } } }
I20260925 14:59:14.619131 30609 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb3f6719b5fd4589889be104691c38ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb3f6719b5fd4589889be104691c38ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42655 } } }
I20260925 14:59:14.619240 30608 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:14.619256 30609 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:14.619580 30617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:14.620110 30617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:14.621646 30617 catalog_manager.cc:1347] Generated new cluster ID: 7656bc9bd90a42f08c16d5cb60ee37a7
I20260925 14:59:14.621691 30617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:14.630645 30617 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:14.630867 30617 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:14.636693 30617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb3f6719b5fd4589889be104691c38ec: Generated new TSK 0
I20260925 14:59:14.636893 30617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:14.639107 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30530
2026-09-25T14:59:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:14 dist-test-slave-1sv7 krb5kdc[30629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:14 dist-test-slave-1sv7 krb5kdc[30629](info): set up 2 sockets
Sep 25 14:59:14 dist-test-slave-1sv7 krb5kdc[30629](info): commencing operation
krb5kdc: starting...
W20260925 14:59:15.700564 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:15 dist-test-slave-1sv7 krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:15.828130 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:15.910391 30642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:15.910640 30642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:15.910682 30642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:15.911983 30642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:15.912060 30642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:15.912083 30642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:15.912113 30642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:15.912142 30642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:15.913726 30642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:15.914011 30642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:15.914252 30642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:15.919282 30650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:15.919484 30642 server_base.cc:1034] running on GCE node
W20260925 14:59:15.919282 30647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:15.919507 30648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:15.919839 30642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:15.920114 30642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:15.921254 30642 hybrid_clock.cc:648] HybridClock initialized: now 1790348355921208 us; error 65 us; skew 500 ppm
Sep 25 14:59:15 dist-test-slave-1sv7 krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:15.930181 30642 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:15.930688 30642 webserver.cc:460] Webserver started at http://127.0.0.1:44767/ using document root <none> and password file <none>
I20260925 14:59:15.930965 30642 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:15.931041 30642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:15.931188 30642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:15.932590 30642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "abbb7a3702e44d258ce22dee1d000401"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-1sv7"
server_key: "ea0b518eb07e906906905db15bbd6c6f"
server_key_iv: "f36dcbd42751e6ba9f646a82acc519fe"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.932960 30642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "abbb7a3702e44d258ce22dee1d000401"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-1sv7"
server_key: "ea0b518eb07e906906905db15bbd6c6f"
server_key_iv: "f36dcbd42751e6ba9f646a82acc519fe"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.934399 30642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:59:15.935330 30657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:15.935551 30642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:15.935662 30642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "abbb7a3702e44d258ce22dee1d000401"
format_stamp: "Formatted at 2026-09-25 14:59:15 on dist-test-slave-1sv7"
server_key: "ea0b518eb07e906906905db15bbd6c6f"
server_key_iv: "f36dcbd42751e6ba9f646a82acc519fe"
server_key_version: "encryptionkey@0"
I20260925 14:59:15.935735 30642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:15.947765 30642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:15.950060 30642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:15.950220 30642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:15.954454 30642 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45033
I20260925 14:59:15.954492 30709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45033 every 8 connection(s)
I20260925 14:59:15.954849 30642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:15.955821 30710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:15.958192 30710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401: Bootstrap starting.
I20260925 14:59:15.958735 30710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:15.958964 30710 log.cc:824] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:15.959659 30710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401: No bootstrap required, opened a new log
I20260925 14:59:15.961320 30710 raft_consensus.cc:348] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbb7a3702e44d258ce22dee1d000401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260925 14:59:15.961467 30710 raft_consensus.cc:374] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:15.961490 30710 raft_consensus.cc:729] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbb7a3702e44d258ce22dee1d000401, State: Initialized, Role: FOLLOWER
I20260925 14:59:15.961623 30710 consensus_queue.cc:260] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbb7a3702e44d258ce22dee1d000401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260925 14:59:15.961709 30710 raft_consensus.cc:388] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:15.961731 30710 raft_consensus.cc:482] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:15.961760 30710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:15.962234 30710 raft_consensus.cc:504] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbb7a3702e44d258ce22dee1d000401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260925 14:59:15.962320 30710 leader_election.cc:302] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abbb7a3702e44d258ce22dee1d000401; no voters:
I20260925 14:59:15.962436 30710 leader_election.cc:288] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:15.962517 30713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:15.962687 30713 raft_consensus.cc:686] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [term 1 LEADER]: Becoming Leader. State: Replica: abbb7a3702e44d258ce22dee1d000401, State: Running, Role: LEADER
I20260925 14:59:15.962728 30710 sys_catalog.cc:564] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:15.962778 30713 consensus_queue.cc:237] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbb7a3702e44d258ce22dee1d000401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } }
I20260925 14:59:15.963416 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30642
I20260925 14:59:15.963388 30714 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abbb7a3702e44d258ce22dee1d000401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbb7a3702e44d258ce22dee1d000401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } } }
I20260925 14:59:15.963482 30714 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:15.963541 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:15.963728 9781 external_mini_cluster.cc:1428] Setting key c0217ba49a54ba432cba779b71974645
I20260925 14:59:15.963856 30722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:15.964010 30715 sys_catalog.cc:455] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abbb7a3702e44d258ce22dee1d000401. Latest consensus state: current_term: 1 leader_uuid: "abbb7a3702e44d258ce22dee1d000401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abbb7a3702e44d258ce22dee1d000401" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45033 } } }
I20260925 14:59:15.964133 30715 sys_catalog.cc:458] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:15.964484 30722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:15.966195 30722 catalog_manager.cc:1347] Generated new cluster ID: f6074d0db23d4d21a6cd3786396ba8f2
I20260925 14:59:15.966246 30722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:59:15 dist-test-slave-1sv7 krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:59:15.976590 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:15.964514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:50875 (local address 127.0.0.1:45033)
0925 14:59:15.964779 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:15.964789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:15.964824 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:59:15.964950 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:15.964958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:15.965020 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:15.965157 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:15.965939 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.967341 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.968440 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.969095 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.969216 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:15.972595 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:15.972600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:15.972600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:15.972614 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:15.974645 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.975140 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.975143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.975143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.975268 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.975513 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.975513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.975514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.975830 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:15.975983 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:15.976178 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:59:15.976292 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260925 14:59:15.977383 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:15.985489 30722 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:15.986164 30722 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:15.986661 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:15.978814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:47777 (local address 127.0.0.1:45033)
0925 14:59:15.978851 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:15.978853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:15.978951 (+ 98us) server_negotiation.cc:407] Connection header received
0925 14:59:15.979033 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:15.979034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:15.979042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:15.979068 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:15.979731 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.980529 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.981521 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:15.981829 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:15.981864 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:15.983376 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:15.983378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:15.983378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:15.983385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:15.985021 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.985690 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.985692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.985692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.985761 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:15.986017 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:15.986018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:15.986018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:15.986163 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:15.986270 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:15.986498 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:59:15.986569 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:15.995666 30722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abbb7a3702e44d258ce22dee1d000401: Generated new TSK 0
I20260925 14:59:15.995879 30722 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:16.002982 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30642
W20260925 14:59:16.007287 30627 connection.cc:535] client connection to 127.0.0.1:45033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45033): Connection reset by peer (error 104)
2026-09-25T14:59:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:16 dist-test-slave-1sv7 krb5kdc[30741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:16 dist-test-slave-1sv7 krb5kdc[30741](info): set up 2 sockets
Sep 25 14:59:16 dist-test-slave-1sv7 krb5kdc[30741](info): commencing operation
krb5kdc: starting...
W20260925 14:59:17.052594 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:17 dist-test-slave-1sv7 krb5kdc[30741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:17.173089 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:17.253495 30754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:17.253746 30754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:17.253784 30754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:17.254935 30754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:17.255002 30754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:17.255026 30754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:17.255046 30754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:17.255077 30754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:17.256567 30754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:17.256836 30754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:17.257066 30754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:17.261973 30759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:17.262022 30762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:17.262001 30760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:17.262301 30754 server_base.cc:1034] running on GCE node
I20260925 14:59:17.262557 30754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:17.262861 30754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:17.264076 30754 hybrid_clock.cc:648] HybridClock initialized: now 1790348357264040 us; error 55 us; skew 500 ppm
Sep 25 14:59:17 dist-test-slave-1sv7 krb5kdc[30741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:17.266681 30754 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:17.267191 30754 webserver.cc:460] Webserver started at http://127.0.0.1:40461/ using document root <none> and password file <none>
I20260925 14:59:17.267416 30754 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:17.267486 30754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:17.267630 30754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:17.268736 30754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "37ef0cd56218466994230e8361e526ab"
format_stamp: "Formatted at 2026-09-25 14:59:17 on dist-test-slave-1sv7"
server_key: "794a4efece697382ba2b3a0e9a94b77b"
server_key_iv: "cf8e24be3d28ca52b1fe357646bd9818"
server_key_version: "encryptionkey@0"
I20260925 14:59:17.269094 30754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "37ef0cd56218466994230e8361e526ab"
format_stamp: "Formatted at 2026-09-25 14:59:17 on dist-test-slave-1sv7"
server_key: "794a4efece697382ba2b3a0e9a94b77b"
server_key_iv: "cf8e24be3d28ca52b1fe357646bd9818"
server_key_version: "encryptionkey@0"
I20260925 14:59:17.270372 30754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:59:17.271095 30769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:17.271260 30754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:59:17.271358 30754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "37ef0cd56218466994230e8361e526ab"
format_stamp: "Formatted at 2026-09-25 14:59:17 on dist-test-slave-1sv7"
server_key: "794a4efece697382ba2b3a0e9a94b77b"
server_key_iv: "cf8e24be3d28ca52b1fe357646bd9818"
server_key_version: "encryptionkey@0"
I20260925 14:59:17.271430 30754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:17.282464 30754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:17.284238 30754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:17.284431 30754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:17.288576 30754 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36635
I20260925 14:59:17.288609 30821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36635 every 8 connection(s)
I20260925 14:59:17.289002 30754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:17.289947 30822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:17.292186 30822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab: Bootstrap starting.
I20260925 14:59:17.292812 30822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:17.293097 30822 log.cc:824] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:17.293877 30822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab: No bootstrap required, opened a new log
I20260925 14:59:17.295540 30822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ef0cd56218466994230e8361e526ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260925 14:59:17.295677 30822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:17.295701 30822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ef0cd56218466994230e8361e526ab, State: Initialized, Role: FOLLOWER
I20260925 14:59:17.295861 30822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ef0cd56218466994230e8361e526ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260925 14:59:17.295925 30822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:17.295946 30822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:17.295974 30822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:17.296600 30822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ef0cd56218466994230e8361e526ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260925 14:59:17.296687 30822 leader_election.cc:302] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37ef0cd56218466994230e8361e526ab; no voters:
I20260925 14:59:17.296882 30822 leader_election.cc:288] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:17.296972 30825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:17.297138 30825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [term 1 LEADER]: Becoming Leader. State: Replica: 37ef0cd56218466994230e8361e526ab, State: Running, Role: LEADER
I20260925 14:59:17.297147 30822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:17.297253 30825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ef0cd56218466994230e8361e526ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260925 14:59:17.297677 30827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37ef0cd56218466994230e8361e526ab. Latest consensus state: current_term: 1 leader_uuid: "37ef0cd56218466994230e8361e526ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ef0cd56218466994230e8361e526ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } } }
I20260925 14:59:17.297821 30827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:17.297900 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30754
I20260925 14:59:17.298038 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:17.298209 30830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:17.298291 9781 external_mini_cluster.cc:1428] Setting key 536064d4e44359a890011024b0be9d51
I20260925 14:59:17.297657 30826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37ef0cd56218466994230e8361e526ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ef0cd56218466994230e8361e526ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } } }
I20260925 14:59:17.298632 30826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:17.298897 30830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:17.300889 30830 catalog_manager.cc:1347] Generated new cluster ID: 70ee51de9b4e4199967e7039e0e6e5b6
I20260925 14:59:17.300943 30830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:17.304682 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:17.298983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:53413 (local address 127.0.0.1:36635)
0925 14:59:17.299308 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:17.299318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:17.299353 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:59:17.299499 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:17.299505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:17.299578 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:17.299733 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:17.299972 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:17.301343 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:17.302430 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:17.303062 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:17.303179 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:17.303925 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:17.303929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:17.303930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:17.304166 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:17.304206 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:17.304221 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:17.304316 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:17.304432 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:59:17.304464 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":68,"threads_started":1}
I20260925 14:59:17.305636 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:17.309194 30830 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:17.309751 30830 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:17.310683 30843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:17.307196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:53513 (local address 127.0.0.1:36635)
0925 14:59:17.307229 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:17.307231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:17.307342 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:59:17.307449 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:17.307450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:17.307461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:17.307496 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:17.307693 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:17.308478 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:17.309193 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:17.309517 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:17.309585 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:17.310447 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:17.310448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:17.310449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:17.310453 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:17.310468 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:17.310472 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:17.310531 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:17.310601 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:59:17.310612 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":17}
I20260925 14:59:17.316251 30830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37ef0cd56218466994230e8361e526ab: Generated new TSK 0
I20260925 14:59:17.316466 30830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:17.324343 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30754
W20260925 14:59:17.328619 30739 connection.cc:535] client connection to 127.0.0.1:36635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36635): Connection reset by peer (error 104)
2026-09-25T14:59:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:17 dist-test-slave-1sv7 krb5kdc[30853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:17 dist-test-slave-1sv7 krb5kdc[30853](info): set up 2 sockets
Sep 25 14:59:17 dist-test-slave-1sv7 krb5kdc[30853](info): commencing operation
krb5kdc: starting...
W20260925 14:59:18.368569 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:18 dist-test-slave-1sv7 krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:18.494879 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:18.577518 30866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:18.577787 30866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:18.577826 30866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:18.579044 30866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:18.579138 30866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:18.579197 30866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:18.579218 30866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:18.579241 30866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:18.580783 30866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:18.580982 30866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:18.581154 30866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:18.586133 30871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:18.586257 30872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:18.586138 30874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:18.586755 30866 server_base.cc:1034] running on GCE node
I20260925 14:59:18.586975 30866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:18.587247 30866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:18.588449 30866 hybrid_clock.cc:648] HybridClock initialized: now 1790348358588424 us; error 31 us; skew 500 ppm
Sep 25 14:59:18 dist-test-slave-1sv7 krb5kdc[30853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:18.591092 30866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:18.591636 30866 webserver.cc:460] Webserver started at http://127.0.0.1:35441/ using document root <none> and password file <none>
I20260925 14:59:18.591908 30866 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:18.591979 30866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:18.592140 30866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:18.593487 30866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "eba5d1c416c54bd082435fa7e03df55f"
format_stamp: "Formatted at 2026-09-25 14:59:18 on dist-test-slave-1sv7"
server_key: "4d4fd9a27e1faf35a94428234b940c56"
server_key_iv: "16fa87114bebf405fcaaf97ec329266c"
server_key_version: "encryptionkey@0"
I20260925 14:59:18.593878 30866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "eba5d1c416c54bd082435fa7e03df55f"
format_stamp: "Formatted at 2026-09-25 14:59:18 on dist-test-slave-1sv7"
server_key: "4d4fd9a27e1faf35a94428234b940c56"
server_key_iv: "16fa87114bebf405fcaaf97ec329266c"
server_key_version: "encryptionkey@0"
I20260925 14:59:18.595252 30866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:18.596127 30881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:18.596352 30866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:18.596503 30866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "eba5d1c416c54bd082435fa7e03df55f"
format_stamp: "Formatted at 2026-09-25 14:59:18 on dist-test-slave-1sv7"
server_key: "4d4fd9a27e1faf35a94428234b940c56"
server_key_iv: "16fa87114bebf405fcaaf97ec329266c"
server_key_version: "encryptionkey@0"
I20260925 14:59:18.596588 30866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:18.599860 30866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:18.601861 30866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:18.602020 30866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:18.606122 30866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37029
I20260925 14:59:18.606151 30933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37029 every 8 connection(s)
I20260925 14:59:18.606491 30866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:18.607515 30934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:18.609843 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30866
I20260925 14:59:18.609998 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:18.610229 9781 external_mini_cluster.cc:1428] Setting key 6765f3885435851f836e020961be267c
I20260925 14:59:18.610425 30934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f: Bootstrap starting.
I20260925 14:59:18.611177 30934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:18.611492 30934 log.cc:824] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:18.612732 30934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f: No bootstrap required, opened a new log
I20260925 14:59:18.613189 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:18.610808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:59197 (local address 127.0.0.1:37029)
0925 14:59:18.611031 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:18.611041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:18.611078 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:59:18.611588 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:18.611597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:18.611635 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:18.611787 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:18.612734 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:18.612737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:18.612738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:18.612742 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:18.612803 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:18.612821 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:18.612914 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:18.612989 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:59:18.613034 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260925 14:59:18.614253 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:18.615017 30934 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba5d1c416c54bd082435fa7e03df55f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260925 14:59:18.615190 30934 raft_consensus.cc:374] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:18.615258 30934 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba5d1c416c54bd082435fa7e03df55f, State: Initialized, Role: FOLLOWER
I20260925 14:59:18.615394 30934 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba5d1c416c54bd082435fa7e03df55f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260925 14:59:18.615507 30934 raft_consensus.cc:388] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:18.615564 30934 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:18.615617 30934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:18.616521 30934 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba5d1c416c54bd082435fa7e03df55f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260925 14:59:18.616636 30934 leader_election.cc:302] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eba5d1c416c54bd082435fa7e03df55f; no voters:
I20260925 14:59:18.616771 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:18.615712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:42777 (local address 127.0.0.1:37029)
0925 14:59:18.615755 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:18.615758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:18.615863 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:59:18.615941 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:18.615942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:18.615946 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:18.615989 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:18.616529 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:18.616530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:18.616530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:18.616533 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:18.616546 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:18.616549 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:18.616622 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:18.616685 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:59:18.616699 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:59:18.616813 30934 leader_election.cc:288] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:18.616855 30944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:18.617077 30944 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [term 1 LEADER]: Becoming Leader. State: Replica: eba5d1c416c54bd082435fa7e03df55f, State: Running, Role: LEADER
I20260925 14:59:18.617164 30934 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:18.617168 30944 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba5d1c416c54bd082435fa7e03df55f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } }
I20260925 14:59:18.617611 30946 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba5d1c416c54bd082435fa7e03df55f. Latest consensus state: current_term: 1 leader_uuid: "eba5d1c416c54bd082435fa7e03df55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba5d1c416c54bd082435fa7e03df55f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } } }
I20260925 14:59:18.617617 30945 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eba5d1c416c54bd082435fa7e03df55f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eba5d1c416c54bd082435fa7e03df55f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37029 } } }
I20260925 14:59:18.617770 30946 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:18.617846 30945 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f [sys.catalog]: This master's current role is: LEADER
W20260925 14:59:18.618897 30960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:18.618959 30960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:18.619011 30953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:18.619196 30953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:18.620740 30953 catalog_manager.cc:1347] Generated new cluster ID: 226882d736f247d7bd50ccf81e4234e8
I20260925 14:59:18.620795 30953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:18.631906 30953 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:18.632093 30953 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:18.641765 30953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba5d1c416c54bd082435fa7e03df55f: Generated new TSK 0
I20260925 14:59:18.642000 30953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:18.650522 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30866
2026-09-25T14:59:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:18 dist-test-slave-1sv7 krb5kdc[30965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:18 dist-test-slave-1sv7 krb5kdc[30965](info): set up 2 sockets
Sep 25 14:59:18 dist-test-slave-1sv7 krb5kdc[30965](info): commencing operation
krb5kdc: starting...
W20260925 14:59:19.716612 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:19 dist-test-slave-1sv7 krb5kdc[30965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:19.835634 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:19.916543 30978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:19.916791 30978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:19.916831 30978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:19.917981 30978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:19.918049 30978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:19.918073 30978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:19.918099 30978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:19.918129 30978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:19.919572 30978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:19.919842 30978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:19.920078 30978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:19.924738 30984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:19.924738 30986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:19.924798 30983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:19.924957 30978 server_base.cc:1034] running on GCE node
I20260925 14:59:19.925200 30978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:19.925467 30978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:19.926640 30978 hybrid_clock.cc:648] HybridClock initialized: now 1790348359926627 us; error 36 us; skew 500 ppm
Sep 25 14:59:19 dist-test-slave-1sv7 krb5kdc[30965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:19.929157 30978 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:19.929567 30978 webserver.cc:460] Webserver started at http://127.0.0.1:34091/ using document root <none> and password file <none>
I20260925 14:59:19.929857 30978 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:19.929929 30978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:19.930088 30978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:19.931115 30978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "636018e141a04da0975ed7b24010fca9"
format_stamp: "Formatted at 2026-09-25 14:59:19 on dist-test-slave-1sv7"
server_key: "174c66671f21e8f7d1d61af797a4e148"
server_key_iv: "8191f760acf9af040f48057087588070"
server_key_version: "encryptionkey@0"
I20260925 14:59:19.931470 30978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "636018e141a04da0975ed7b24010fca9"
format_stamp: "Formatted at 2026-09-25 14:59:19 on dist-test-slave-1sv7"
server_key: "174c66671f21e8f7d1d61af797a4e148"
server_key_iv: "8191f760acf9af040f48057087588070"
server_key_version: "encryptionkey@0"
I20260925 14:59:19.932854 30978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:59:19.933661 30993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:19.933914 30978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:19.934031 30978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "636018e141a04da0975ed7b24010fca9"
format_stamp: "Formatted at 2026-09-25 14:59:19 on dist-test-slave-1sv7"
server_key: "174c66671f21e8f7d1d61af797a4e148"
server_key_iv: "8191f760acf9af040f48057087588070"
server_key_version: "encryptionkey@0"
I20260925 14:59:19.934109 30978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:19.943529 30978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:19.945654 30978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:19.945816 30978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:19.949731 30978 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38857
I20260925 14:59:19.949791 31045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38857 every 8 connection(s)
I20260925 14:59:19.950139 30978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:19.950497 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 30978
I20260925 14:59:19.950627 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:19.950834 9781 external_mini_cluster.cc:1428] Setting key 3d664c4d350bc2ddfbfc30ddbd8ecb62
I20260925 14:59:19.951610 31046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:19.954577 31046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9: Bootstrap starting.
I20260925 14:59:19.955273 31046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:19.955549 31046 log.cc:824] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:19.956243 31046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9: No bootstrap required, opened a new log
Sep 25 14:59:19 dist-test-slave-1sv7 krb5kdc[30965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:59:19.958410 31046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636018e141a04da0975ed7b24010fca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260925 14:59:19.958539 31046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:19.958563 31046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636018e141a04da0975ed7b24010fca9, State: Initialized, Role: FOLLOWER
I20260925 14:59:19.958693 31046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636018e141a04da0975ed7b24010fca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260925 14:59:19.958801 31046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:19.958832 31046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:19.958879 31046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:19.959443 31046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636018e141a04da0975ed7b24010fca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260925 14:59:19.959554 31046 leader_election.cc:302] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 636018e141a04da0975ed7b24010fca9; no voters:
I20260925 14:59:19.959728 31046 leader_election.cc:288] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:19.959785 31051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:19.959980 31051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [term 1 LEADER]: Becoming Leader. State: Replica: 636018e141a04da0975ed7b24010fca9, State: Running, Role: LEADER
I20260925 14:59:19.959987 31046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:19.960135 31051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636018e141a04da0975ed7b24010fca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } }
I20260925 14:59:19.960536 31051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 636018e141a04da0975ed7b24010fca9. Latest consensus state: current_term: 1 leader_uuid: "636018e141a04da0975ed7b24010fca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636018e141a04da0975ed7b24010fca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } } }
I20260925 14:59:19.961141 31051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:19.961378 31053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "636018e141a04da0975ed7b24010fca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636018e141a04da0975ed7b24010fca9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38857 } } }
I20260925 14:59:19.961445 31062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:19.962080 31062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:19.961566 31053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:19.962525 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:19.951484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:57379 (local address 127.0.0.1:38857)
0925 14:59:19.951677 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:19.951686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:19.951731 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:59:19.952190 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:19.952197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:19.952243 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:19.952438 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:19.953187 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.954109 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.955186 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.955636 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.955747 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.958419 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:19.958423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:19.958423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:19.958435 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:19.960570 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:19.961051 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:19.961054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:19.961054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:19.961155 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:19.961473 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:19.961474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:19.961474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:19.961782 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:19.961963 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:19.962307 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:59:19.962386 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260925 14:59:19.963474 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:19.963588 31062 catalog_manager.cc:1347] Generated new cluster ID: 16e5d1c02dcb49b3a795dc42d8f6a449
I20260925 14:59:19.963629 31062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:19.972044 31049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:19.964914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:38669 (local address 127.0.0.1:38857)
0925 14:59:19.964950 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:19.964952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:19.965080 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:59:19.965138 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:19.965139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:19.965148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:19.965198 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:19.965809 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.966518 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.967511 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:19.967823 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:19.967881 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:19.969308 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:19.969309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:19.969309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:19.969315 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:19.970829 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:19.971239 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:19.971240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:19.971240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:19.971299 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:19.971532 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:19.971533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:19.971533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:19.971652 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:19.971742 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:19.971901 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:59:19.971960 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:19.986541 31062 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:19.987289 31062 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:19.994431 31062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 636018e141a04da0975ed7b24010fca9: Generated new TSK 0
I20260925 14:59:19.994609 31062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:19.997159 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 30978
W20260925 14:59:20.001361 30963 connection.cc:535] client connection to 127.0.0.1:38857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38857): Connection reset by peer (error 104)
2026-09-25T14:59:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:20 dist-test-slave-1sv7 krb5kdc[31077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:20 dist-test-slave-1sv7 krb5kdc[31077](info): set up 2 sockets
Sep 25 14:59:20 dist-test-slave-1sv7 krb5kdc[31077](info): commencing operation
krb5kdc: starting...
W20260925 14:59:21.033077 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:21 dist-test-slave-1sv7 krb5kdc[31077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:21.154955 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43637
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:21.234633 31090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:21.234845 31090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:21.234870 31090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:21.235981 31090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:21.236016 31090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:21.236030 31090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:21.236042 31090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:21.236054 31090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:21.237479 31090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43637
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:21.237669 31090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:21.237843 31090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:21.242642 31098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:21.242707 31096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:21.242642 31095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:21.243155 31090 server_base.cc:1034] running on GCE node
I20260925 14:59:21.243423 31090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:21.243733 31090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:21.244918 31090 hybrid_clock.cc:648] HybridClock initialized: now 1790348361244889 us; error 47 us; skew 500 ppm
Sep 25 14:59:21 dist-test-slave-1sv7 krb5kdc[31077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:21.251068 31090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:21.251562 31090 webserver.cc:460] Webserver started at http://127.0.0.1:43443/ using document root <none> and password file <none>
I20260925 14:59:21.251816 31090 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:21.251884 31090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:21.252038 31090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:21.253221 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "a0ce66441f1742c6befa65ded32bb6b3"
format_stamp: "Formatted at 2026-09-25 14:59:21 on dist-test-slave-1sv7"
server_key: "09017b63f789425edc733720ec3c8ec8"
server_key_iv: "f966f86f1b938a93159e6391c177bd3e"
server_key_version: "encryptionkey@0"
I20260925 14:59:21.253589 31090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "a0ce66441f1742c6befa65ded32bb6b3"
format_stamp: "Formatted at 2026-09-25 14:59:21 on dist-test-slave-1sv7"
server_key: "09017b63f789425edc733720ec3c8ec8"
server_key_iv: "f966f86f1b938a93159e6391c177bd3e"
server_key_version: "encryptionkey@0"
I20260925 14:59:21.254968 31090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:59:21.255817 31105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:21.256018 31090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:21.256125 31090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "a0ce66441f1742c6befa65ded32bb6b3"
format_stamp: "Formatted at 2026-09-25 14:59:21 on dist-test-slave-1sv7"
server_key: "09017b63f789425edc733720ec3c8ec8"
server_key_iv: "f966f86f1b938a93159e6391c177bd3e"
server_key_version: "encryptionkey@0"
I20260925 14:59:21.256210 31090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:21.278143 31090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:21.280566 31090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:21.280737 31090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:21.284559 31090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43637
I20260925 14:59:21.284633 31157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43637 every 8 connection(s)
I20260925 14:59:21.284984 31090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:21.285997 31158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:21.288259 31158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3: Bootstrap starting.
I20260925 14:59:21.288861 31158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:21.289096 31158 log.cc:824] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:21.289808 31158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3: No bootstrap required, opened a new log
I20260925 14:59:21.289994 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31090
I20260925 14:59:21.290131 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:21.290337 9781 external_mini_cluster.cc:1428] Setting key 232b5149dda36874f6591d0ac616a4e2
I20260925 14:59:21.292182 31158 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce66441f1742c6befa65ded32bb6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260925 14:59:21.292347 31158 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:21.292387 31158 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ce66441f1742c6befa65ded32bb6b3, State: Initialized, Role: FOLLOWER
I20260925 14:59:21.292544 31158 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce66441f1742c6befa65ded32bb6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260925 14:59:21.292640 31158 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:21.292692 31158 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:21.292743 31158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:21.293290 31158 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce66441f1742c6befa65ded32bb6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260925 14:59:21.293411 31158 leader_election.cc:302] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ce66441f1742c6befa65ded32bb6b3; no voters:
I20260925 14:59:21.293579 31158 leader_election.cc:288] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:21.293644 31163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:21.293800 31163 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [term 1 LEADER]: Becoming Leader. State: Replica: a0ce66441f1742c6befa65ded32bb6b3, State: Running, Role: LEADER
I20260925 14:59:21.293852 31158 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:21.293929 31163 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce66441f1742c6befa65ded32bb6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } }
I20260925 14:59:21.294314 31164 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0ce66441f1742c6befa65ded32bb6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce66441f1742c6befa65ded32bb6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } } }
I20260925 14:59:21.294401 31164 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:21.294303 31163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0ce66441f1742c6befa65ded32bb6b3. Latest consensus state: current_term: 1 leader_uuid: "a0ce66441f1742c6befa65ded32bb6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ce66441f1742c6befa65ded32bb6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43637 } } }
I20260925 14:59:21.294526 31163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:21.295202 31172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:21.295874 31172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:21.296922 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:21.290864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:58835 (local address 127.0.0.1:43637)
0925 14:59:21.291043 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:21.291050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:21.291075 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:59:21.291625 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:21.291632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:21.291683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:21.291796 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:21.292044 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.293371 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.294723 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.295360 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.295477 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.296202 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:21.296206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:21.296206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:21.296431 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:21.296475 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:21.296486 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:21.296582 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:21.296665 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:59:21.296694 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 14:59:21.297493 31172 catalog_manager.cc:1347] Generated new cluster ID: 0f30156fae2343d7858681cb4085f4cd
I20260925 14:59:21.297544 31172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:21.300701 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:21.306540 31172 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:21.307224 31172 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:21.309489 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:21.302143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:55737 (local address 127.0.0.1:43637)
0925 14:59:21.302179 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:21.302181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:21.304615 (+ 2434us) server_negotiation.cc:407] Connection header received
0925 14:59:21.304693 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:21.304693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:21.304704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:21.304767 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:21.304917 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.305440 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.306197 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:21.306469 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:21.306550 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:21.309187 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:21.309188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:21.309189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:21.309193 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:21.309207 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:21.309211 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:21.309263 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:21.309382 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 14:59:21.309393 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":18}
I20260925 14:59:21.312119 31172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0ce66441f1742c6befa65ded32bb6b3: Generated new TSK 0
I20260925 14:59:21.312286 31172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:21.316320 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31090
W20260925 14:59:21.321102 31075 connection.cc:535] client connection to 127.0.0.1:43637 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43637): Connection reset by peer (error 104)
2026-09-25T14:59:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:21 dist-test-slave-1sv7 krb5kdc[31189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:21 dist-test-slave-1sv7 krb5kdc[31189](info): set up 2 sockets
Sep 25 14:59:21 dist-test-slave-1sv7 krb5kdc[31189](info): commencing operation
krb5kdc: starting...
W20260925 14:59:22.412587 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:22 dist-test-slave-1sv7 krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:59:22.533421 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:22.619329 31202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:22.619549 31202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:22.619575 31202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:22.620738 31202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:22.620779 31202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:22.620793 31202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:22.620806 31202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:22.620818 31202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:22.622254 31202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:22.622458 31202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:22.622637 31202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:22.627882 31208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:22.627882 31210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:22.627915 31202 server_base.cc:1034] running on GCE node
W20260925 14:59:22.627900 31207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:22.628533 31202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:22.628860 31202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:22.630074 31202 hybrid_clock.cc:648] HybridClock initialized: now 1790348362630056 us; error 28 us; skew 500 ppm
Sep 25 14:59:22 dist-test-slave-1sv7 krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:22.633029 31202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:59:22.633558 31202 webserver.cc:460] Webserver started at http://127.0.0.1:35993/ using document root <none> and password file <none>
I20260925 14:59:22.633803 31202 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:22.633893 31202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:22.634052 31202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:22.635192 31202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "5b6a367d78a04a3f9c45d7c1f24edb95"
format_stamp: "Formatted at 2026-09-25 14:59:22 on dist-test-slave-1sv7"
server_key: "fcf9136fd98d7dfd717ff63926772b1b"
server_key_iv: "2e1ad618754826ea2b7b619a77ecdc48"
server_key_version: "encryptionkey@0"
I20260925 14:59:22.635579 31202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "5b6a367d78a04a3f9c45d7c1f24edb95"
format_stamp: "Formatted at 2026-09-25 14:59:22 on dist-test-slave-1sv7"
server_key: "fcf9136fd98d7dfd717ff63926772b1b"
server_key_iv: "2e1ad618754826ea2b7b619a77ecdc48"
server_key_version: "encryptionkey@0"
I20260925 14:59:22.637116 31202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:22.638090 31217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:22.638422 31202 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:22.638568 31202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "5b6a367d78a04a3f9c45d7c1f24edb95"
format_stamp: "Formatted at 2026-09-25 14:59:22 on dist-test-slave-1sv7"
server_key: "fcf9136fd98d7dfd717ff63926772b1b"
server_key_iv: "2e1ad618754826ea2b7b619a77ecdc48"
server_key_version: "encryptionkey@0"
I20260925 14:59:22.638659 31202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:22.642441 31202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:22.644912 31202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:22.645078 31202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:22.649382 31202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41111
I20260925 14:59:22.649442 31269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41111 every 8 connection(s)
I20260925 14:59:22.649804 31202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:22.650928 31270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:22.653522 31270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95: Bootstrap starting.
I20260925 14:59:22.654110 31270 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:22.654377 31270 log.cc:824] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:22.655089 31270 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95: No bootstrap required, opened a new log
I20260925 14:59:22.656906 31270 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260925 14:59:22.657054 31270 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:22.657079 31270 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6a367d78a04a3f9c45d7c1f24edb95, State: Initialized, Role: FOLLOWER
I20260925 14:59:22.657235 31270 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260925 14:59:22.657320 31270 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:22.657343 31270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:22.657373 31270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:22.657895 31270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260925 14:59:22.658002 31270 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6a367d78a04a3f9c45d7c1f24edb95; no voters:
I20260925 14:59:22.658136 31270 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:22.658245 31273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:22.658428 31273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6a367d78a04a3f9c45d7c1f24edb95, State: Running, Role: LEADER
I20260925 14:59:22.658444 31270 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:22.658666 31273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } }
I20260925 14:59:22.659072 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31202
I20260925 14:59:22.659215 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:22.659271 31275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6a367d78a04a3f9c45d7c1f24edb95. Latest consensus state: current_term: 1 leader_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } } }
I20260925 14:59:22.659341 31275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:22.659444 9781 external_mini_cluster.cc:1428] Setting key d6d33945f3a757d75b55dc130c5d0131
I20260925 14:59:22.659461 31274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6a367d78a04a3f9c45d7c1f24edb95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41111 } } }
I20260925 14:59:22.659520 31274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:22.659595 31282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:22.660126 31282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:22.662040 31282 catalog_manager.cc:1347] Generated new cluster ID: 2f4f559b866f447da00a6b74d8391d19
I20260925 14:59:22.662091 31282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:22.662639 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:22.659946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:39929 (local address 127.0.0.1:41111)
0925 14:59:22.661000 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:22.661010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:22.661043 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:59:22.661159 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:22.661167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:22.661198 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:22.661378 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:22.662129 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:22.662133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:22.662133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:22.662138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:22.662188 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:22.662201 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:22.662282 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:22.662413 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 14:59:22.662467 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":75,"threads_started":1}
I20260925 14:59:22.663544 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:22.669793 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:22.665063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:59687 (local address 127.0.0.1:41111)
0925 14:59:22.665102 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:22.665104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:22.668608 (+ 3504us) server_negotiation.cc:407] Connection header received
0925 14:59:22.668739 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:22.668740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:22.668746 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:22.668823 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:22.669517 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:22.669519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:22.669519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:22.669524 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:22.669541 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:22.669545 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:22.669609 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:22.669679 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:59:22.669695 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:22.679001 31282 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:22.679212 31282 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:22.681439 31282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6a367d78a04a3f9c45d7c1f24edb95: Generated new TSK 0
I20260925 14:59:22.681649 31282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:22.689435 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31202
2026-09-25T14:59:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:22 dist-test-slave-1sv7 krb5kdc[31301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:22 dist-test-slave-1sv7 krb5kdc[31301](info): set up 2 sockets
Sep 25 14:59:22 dist-test-slave-1sv7 krb5kdc[31301](info): commencing operation
krb5kdc: starting...
W20260925 14:59:23.729041 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:23 dist-test-slave-1sv7 krb5kdc[31301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:23.857148 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39733
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:23.941366 31314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:23.941661 31314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:23.941700 31314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:23.942898 31314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:23.942941 31314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:23.942955 31314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:23.942970 31314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:23.942981 31314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:23.944531 31314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:39733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:23.944796 31314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:23.945047 31314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:23.950171 31319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:23.950287 31314 server_base.cc:1034] running on GCE node
W20260925 14:59:23.950212 31322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:23.950165 31320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:23.950778 31314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:23.951133 31314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:23.952306 31314 hybrid_clock.cc:648] HybridClock initialized: now 1790348363952276 us; error 48 us; skew 500 ppm
Sep 25 14:59:23 dist-test-slave-1sv7 krb5kdc[31301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348363, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:23.955029 31314 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:23.955566 31314 webserver.cc:460] Webserver started at http://127.9.141.126:42243/ using document root <none> and password file <none>
I20260925 14:59:23.955865 31314 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:23.955940 31314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:23.956094 31314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:23.957257 31314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "32f006badef24aa4be4f870f942b3e88"
format_stamp: "Formatted at 2026-09-25 14:59:23 on dist-test-slave-1sv7"
server_key: "3872e52d3ff88db3af02b0fc1b9f4fb9"
server_key_iv: "63546c592044bf2f1614b26b7d0cfe3b"
server_key_version: "encryptionkey@0"
I20260925 14:59:23.957619 31314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "32f006badef24aa4be4f870f942b3e88"
format_stamp: "Formatted at 2026-09-25 14:59:23 on dist-test-slave-1sv7"
server_key: "3872e52d3ff88db3af02b0fc1b9f4fb9"
server_key_iv: "63546c592044bf2f1614b26b7d0cfe3b"
server_key_version: "encryptionkey@0"
I20260925 14:59:23.959029 31314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:59:23.959889 31329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:23.960089 31314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:59:23.960197 31314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "32f006badef24aa4be4f870f942b3e88"
format_stamp: "Formatted at 2026-09-25 14:59:23 on dist-test-slave-1sv7"
server_key: "3872e52d3ff88db3af02b0fc1b9f4fb9"
server_key_iv: "63546c592044bf2f1614b26b7d0cfe3b"
server_key_version: "encryptionkey@0"
I20260925 14:59:23.960278 31314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:23.988091 31314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:23.990361 31314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:23.990536 31314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:23.994841 31314 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:39733
I20260925 14:59:23.994889 31381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:39733 every 8 connection(s)
I20260925 14:59:23.995253 31314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:23.996189 31382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:23.998522 31382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88: Bootstrap starting.
I20260925 14:59:23.999162 31382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:23.999470 31382 log.cc:824] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:24.000290 31382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88: No bootstrap required, opened a new log
I20260925 14:59:24.001955 31382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f006badef24aa4be4f870f942b3e88" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39733 } }
I20260925 14:59:24.002097 31382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:24.002130 31382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f006badef24aa4be4f870f942b3e88, State: Initialized, Role: FOLLOWER
I20260925 14:59:24.002313 31382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f006badef24aa4be4f870f942b3e88" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39733 } }
I20260925 14:59:24.002378 31382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:24.002400 31382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:24.002475 31382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:24.002692 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31314
I20260925 14:59:24.002818 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:24.003017 9781 external_mini_cluster.cc:1428] Setting key 1258cf0715d2a79985289ad631b56593
I20260925 14:59:24.003636 31382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f006badef24aa4be4f870f942b3e88" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39733 } }
I20260925 14:59:24.003779 31382 leader_election.cc:302] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32f006badef24aa4be4f870f942b3e88; no voters:
I20260925 14:59:24.003952 31382 leader_election.cc:288] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:24.004014 31387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:24.004181 31387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [term 1 LEADER]: Becoming Leader. State: Replica: 32f006badef24aa4be4f870f942b3e88, State: Running, Role: LEADER
I20260925 14:59:24.004216 31382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:24.004309 31387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f006badef24aa4be4f870f942b3e88" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39733 } }
I20260925 14:59:24.004719 31387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32f006badef24aa4be4f870f942b3e88. Latest consensus state: current_term: 1 leader_uuid: "32f006badef24aa4be4f870f942b3e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f006badef24aa4be4f870f942b3e88" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39733 } } }
I20260925 14:59:24.004725 31388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32f006badef24aa4be4f870f942b3e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32f006badef24aa4be4f870f942b3e88" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 39733 } } }
I20260925 14:59:24.004812 31387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:24.004842 31388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:24.005105 31396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:24.005795 31396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:24.007475 31396 catalog_manager.cc:1347] Generated new cluster ID: aad28ba836a24646b5ce50d2d438a29c
I20260925 14:59:24.007532 31396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:59:24 dist-test-slave-1sv7 krb5kdc[31301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:24.015180 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:24.003572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33411 (local address 127.9.141.126:39733)
0925 14:59:24.003833 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:24.003844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:24.003880 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:59:24.004010 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:24.004018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:24.004083 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:24.005203 (+ 1120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:24.005337 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.006834 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.007822 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.008255 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.008378 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.010808 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:24.010812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:24.010813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:24.010825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:24.013211 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:24.013706 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:24.013709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:24.013710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:24.013819 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:24.014056 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:24.014058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:24.014058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:24.014345 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:24.014547 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:24.014849 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 14:59:24.014935 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
I20260925 14:59:24.017196 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:24.020426 31396 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:24.021063 31396 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:24.027117 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:24.018575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45159 (local address 127.9.141.126:39733)
0925 14:59:24.018624 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:24.018626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:24.019959 (+ 1333us) server_negotiation.cc:407] Connection header received
0925 14:59:24.020004 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:24.020005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:24.020015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:24.020067 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:24.020755 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.021499 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.022540 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:24.022839 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:24.022900 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:24.024327 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:24.024329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:24.024329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:24.024336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:24.025878 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:24.026300 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:24.026301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:24.026301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:24.026356 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:24.026582 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:24.026583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:24.026583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:24.026699 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:24.026805 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:24.026958 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:59:24.027017 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:59:24.030859 31396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32f006badef24aa4be4f870f942b3e88: Generated new TSK 0
I20260925 14:59:24.031045 31396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:24.034847 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31314
W20260925 14:59:24.039027 31299 connection.cc:535] client connection to 127.9.141.126:39733 recv error: Network error: failed to read from TLS socket (remote: 127.9.141.126:39733): Connection reset by peer (error 104)
2026-09-25T14:59:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:24 dist-test-slave-1sv7 krb5kdc[31413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:24 dist-test-slave-1sv7 krb5kdc[31413](info): set up 2 sockets
Sep 25 14:59:24 dist-test-slave-1sv7 krb5kdc[31413](info): commencing operation
krb5kdc: starting...
W20260925 14:59:25.072638 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:25 dist-test-slave-1sv7 krb5kdc[31413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:25.195580 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:42853
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:42853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:25.278028 31426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:25.278304 31426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:25.278347 31426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:25.279587 31426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:25.279677 31426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:25.279704 31426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:25.279735 31426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:25.279765 31426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:25.281915 31426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:42853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:42853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:25.282222 31426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:25.282459 31426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:25.287552 31431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:25.287583 31432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:25.287497 31434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:25.287819 31426 server_base.cc:1034] running on GCE node
I20260925 14:59:25.288095 31426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:25.288369 31426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:25.289620 31426 hybrid_clock.cc:648] HybridClock initialized: now 1790348365289574 us; error 62 us; skew 500 ppm
Sep 25 14:59:25 dist-test-slave-1sv7 krb5kdc[31413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348365, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:25.292541 31426 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:25.293150 31426 webserver.cc:460] Webserver started at http://127.9.141.126:39589/ using document root <none> and password file <none>
I20260925 14:59:25.293404 31426 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:25.293475 31426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:25.293627 31426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:25.294737 31426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "b53368e4145042c6a398228c62fa568c"
format_stamp: "Formatted at 2026-09-25 14:59:25 on dist-test-slave-1sv7"
server_key: "aff94ac0ba4944d5de509c356ab8fd7d"
server_key_iv: "6c8e9637ddf1db5100789e36471f0253"
server_key_version: "encryptionkey@0"
I20260925 14:59:25.295112 31426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "b53368e4145042c6a398228c62fa568c"
format_stamp: "Formatted at 2026-09-25 14:59:25 on dist-test-slave-1sv7"
server_key: "aff94ac0ba4944d5de509c356ab8fd7d"
server_key_iv: "6c8e9637ddf1db5100789e36471f0253"
server_key_version: "encryptionkey@0"
I20260925 14:59:25.296738 31426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:25.297641 31441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:25.297863 31426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:25.297979 31426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "b53368e4145042c6a398228c62fa568c"
format_stamp: "Formatted at 2026-09-25 14:59:25 on dist-test-slave-1sv7"
server_key: "aff94ac0ba4944d5de509c356ab8fd7d"
server_key_iv: "6c8e9637ddf1db5100789e36471f0253"
server_key_version: "encryptionkey@0"
I20260925 14:59:25.298070 31426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:25.312831 31426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:25.315193 31426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:25.315317 31426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:25.320996 31426 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:42853
I20260925 14:59:25.321038 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:42853 every 8 connection(s)
I20260925 14:59:25.321442 31426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:25.322821 31494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:25.325871 31494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c: Bootstrap starting.
I20260925 14:59:25.326858 31494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:25.327205 31494 log.cc:824] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:25.328297 31494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c: No bootstrap required, opened a new log
I20260925 14:59:25.331068 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31426
I20260925 14:59:25.331214 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:25.331046 31494 raft_consensus.cc:348] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53368e4145042c6a398228c62fa568c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42853 } }
I20260925 14:59:25.331276 31494 raft_consensus.cc:374] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:25.331321 31494 raft_consensus.cc:729] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b53368e4145042c6a398228c62fa568c, State: Initialized, Role: FOLLOWER
I20260925 14:59:25.331430 9781 external_mini_cluster.cc:1428] Setting key 85d360ea90636efff47ab61f4092d757
I20260925 14:59:25.331482 31494 consensus_queue.cc:260] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53368e4145042c6a398228c62fa568c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42853 } }
I20260925 14:59:25.331874 31494 raft_consensus.cc:388] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:25.331923 31494 raft_consensus.cc:482] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:25.331975 31494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:25.333499 31494 raft_consensus.cc:504] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53368e4145042c6a398228c62fa568c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42853 } }
I20260925 14:59:25.333652 31494 leader_election.cc:302] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b53368e4145042c6a398228c62fa568c; no voters:
I20260925 14:59:25.333847 31494 leader_election.cc:288] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:25.333907 31499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:25.334059 31499 raft_consensus.cc:686] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [term 1 LEADER]: Becoming Leader. State: Replica: b53368e4145042c6a398228c62fa568c, State: Running, Role: LEADER
I20260925 14:59:25.334131 31494 sys_catalog.cc:564] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:25.334151 31499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53368e4145042c6a398228c62fa568c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42853 } }
I20260925 14:59:25.334543 31500 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b53368e4145042c6a398228c62fa568c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53368e4145042c6a398228c62fa568c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42853 } } }
I20260925 14:59:25.334558 31501 sys_catalog.cc:455] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b53368e4145042c6a398228c62fa568c. Latest consensus state: current_term: 1 leader_uuid: "b53368e4145042c6a398228c62fa568c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b53368e4145042c6a398228c62fa568c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 42853 } } }
I20260925 14:59:25.334728 31500 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:25.334748 31501 sys_catalog.cc:458] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:25.335143 31508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:25.335901 31508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:25.337687 31508 catalog_manager.cc:1347] Generated new cluster ID: 3b643c2b937543018b5b7a576b27aad4
I20260925 14:59:25.337749 31508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:25.341441 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:25.332183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59105 (local address 127.9.141.126:42853)
0925 14:59:25.332422 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:25.332434 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:25.335856 (+ 3422us) server_negotiation.cc:407] Connection header received
0925 14:59:25.336044 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:25.336054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:25.336118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:25.336279 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:25.336610 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.337987 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.339137 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.339781 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.339890 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.340726 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:25.340730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:25.340731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:25.340949 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:25.340989 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:25.341004 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:25.341085 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:25.341198 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:59:25.341226 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260925 14:59:25.342705 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:25.347905 31498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:25.344194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44811 (local address 127.9.141.126:42853)
0925 14:59:25.344229 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:25.344231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:25.344341 (+ 110us) server_negotiation.cc:407] Connection header received
0925 14:59:25.344440 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:25.344441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:25.344452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:25.344513 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:25.344671 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.345438 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.346486 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:25.346796 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:25.346855 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:25.347627 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:25.347629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:25.347629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:25.347642 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:25.347657 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:25.347662 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:25.347717 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:25.347809 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:59:25.347822 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":17}
I20260925 14:59:25.357105 31508 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:25.358044 31508 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:25.364732 31508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b53368e4145042c6a398228c62fa568c: Generated new TSK 0
I20260925 14:59:25.365010 31508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:25.369563 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31426
W20260925 14:59:25.375871 31411 connection.cc:535] client connection to 127.9.141.126:42853 recv error: Network error: failed to read from TLS socket (remote: 127.9.141.126:42853): Connection reset by peer (error 104)
2026-09-25T14:59:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:25 dist-test-slave-1sv7 krb5kdc[31525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:25 dist-test-slave-1sv7 krb5kdc[31525](info): set up 2 sockets
Sep 25 14:59:25 dist-test-slave-1sv7 krb5kdc[31525](info): commencing operation
krb5kdc: starting...
W20260925 14:59:26.433029 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:26 dist-test-slave-1sv7 krb5kdc[31525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:26.568357 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36931
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:39527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:26.650182 31538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:26.650393 31538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:26.650416 31538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:26.651595 31538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:26.651640 31538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:26.651654 31538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:26.651667 31538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:26.651679 31538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:26.653399 31538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:39527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:26.653657 31538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:26.653914 31538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:26.659626 31546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:26.659631 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:26.659613 31543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:26.659835 31538 server_base.cc:1034] running on GCE node
I20260925 14:59:26.660163 31538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:26.660573 31538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:26.661790 31538 hybrid_clock.cc:648] HybridClock initialized: now 1790348366661778 us; error 50 us; skew 500 ppm
Sep 25 14:59:26 dist-test-slave-1sv7 krb5kdc[31525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348366, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:26.669199 31538 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:26.670028 31538 webserver.cc:460] Webserver started at http://127.9.141.126:41843/ using document root <none> and password file <none>
I20260925 14:59:26.670351 31538 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:26.670428 31538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:26.670603 31538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:26.672042 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "c42a22c60fc1494aade8a4ca7e768d7c"
format_stamp: "Formatted at 2026-09-25 14:59:26 on dist-test-slave-1sv7"
server_key: "76a2037fbc6ee89d2678a624b00d5052"
server_key_iv: "86d0f74c0e78bb03216ecd016b705c2e"
server_key_version: "encryptionkey@0"
I20260925 14:59:26.672559 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "c42a22c60fc1494aade8a4ca7e768d7c"
format_stamp: "Formatted at 2026-09-25 14:59:26 on dist-test-slave-1sv7"
server_key: "76a2037fbc6ee89d2678a624b00d5052"
server_key_iv: "86d0f74c0e78bb03216ecd016b705c2e"
server_key_version: "encryptionkey@0"
I20260925 14:59:26.674198 31538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:26.675308 31553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:26.675746 31538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:26.675876 31538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "c42a22c60fc1494aade8a4ca7e768d7c"
format_stamp: "Formatted at 2026-09-25 14:59:26 on dist-test-slave-1sv7"
server_key: "76a2037fbc6ee89d2678a624b00d5052"
server_key_iv: "86d0f74c0e78bb03216ecd016b705c2e"
server_key_version: "encryptionkey@0"
I20260925 14:59:26.675959 31538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:26.679510 31538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:26.681578 31538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:26.681756 31538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:26.686048 31538 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:36931
I20260925 14:59:26.686081 31605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:36931 every 8 connection(s)
I20260925 14:59:26.686419 31538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:26.687438 31606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:26.689949 31606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c: Bootstrap starting.
I20260925 14:59:26.690549 31606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:26.690851 31606 log.cc:824] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:26.691618 31606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c: No bootstrap required, opened a new log
I20260925 14:59:26.693415 31606 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36931 } }
I20260925 14:59:26.693565 31606 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:26.693588 31606 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42a22c60fc1494aade8a4ca7e768d7c, State: Initialized, Role: FOLLOWER
I20260925 14:59:26.693640 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31538
I20260925 14:59:26.693776 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:26.693745 31606 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36931 } }
I20260925 14:59:26.693847 31606 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:26.693888 31606 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:26.693961 31606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:26.693974 9781 external_mini_cluster.cc:1428] Setting key 5c8829559644c2b70c528c0e9a277a78
I20260925 14:59:26.695120 31606 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36931 } }
I20260925 14:59:26.695250 31606 leader_election.cc:302] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42a22c60fc1494aade8a4ca7e768d7c; no voters:
I20260925 14:59:26.695423 31606 leader_election.cc:288] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:26.695487 31611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:26.695618 31611 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [term 1 LEADER]: Becoming Leader. State: Replica: c42a22c60fc1494aade8a4ca7e768d7c, State: Running, Role: LEADER
I20260925 14:59:26.695705 31606 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:26.695709 31611 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36931 } }
I20260925 14:59:26.696142 31612 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36931 } } }
I20260925 14:59:26.696225 31612 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:26.696424 31613 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42a22c60fc1494aade8a4ca7e768d7c. Latest consensus state: current_term: 1 leader_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42a22c60fc1494aade8a4ca7e768d7c" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36931 } } }
I20260925 14:59:26.696507 31613 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:26.696913 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:26.694582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55553 (local address 127.9.141.126:36931)
0925 14:59:26.694765 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:26.694773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:26.694805 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:59:26.694901 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:26.694907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:26.694931 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:26.695047 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:26.696088 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:26.696092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:26.696092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:26.696097 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:26.696148 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:26.696164 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:26.696234 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:26.696678 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 14:59:26.696725 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 14:59:26.697091 31618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:26.697592 31618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:26.698055 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:26.700470 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:26.699484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57991 (local address 127.9.141.126:36931)
0925 14:59:26.699520 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:26.699522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:26.699613 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:59:26.699675 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:26.699676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:26.699680 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:26.699725 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:26.700238 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:26.700239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:26.700240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:26.700243 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:26.700254 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:26.700258 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:26.700298 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:26.700356 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:59:26.700367 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:26.704855 31618 catalog_manager.cc:1347] Generated new cluster ID: 804d61a797a9494a964d32972d680d79
I20260925 14:59:26.704918 31618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:26.716468 31618 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:26.716675 31618 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:26.725030 31618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c42a22c60fc1494aade8a4ca7e768d7c: Generated new TSK 0
I20260925 14:59:26.725247 31618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:26.730342 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31538
2026-09-25T14:59:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:26 dist-test-slave-1sv7 krb5kdc[31637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:26 dist-test-slave-1sv7 krb5kdc[31637](info): set up 2 sockets
Sep 25 14:59:26 dist-test-slave-1sv7 krb5kdc[31637](info): commencing operation
krb5kdc: starting...
W20260925 14:59:27.780586 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:27 dist-test-slave-1sv7 krb5kdc[31637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:27.911255 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35901
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:38867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:35901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:27.997047 31650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:27.997269 31650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:27.997293 31650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:27.998418 31650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:27.998461 31650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:27.998476 31650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:27.998487 31650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:27.998500 31650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:27.999940 31650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:38867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:35901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:35901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:28.000151 31650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:28.000330 31650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:59:28.005529 31650 server_base.cc:1034] running on GCE node
W20260925 14:59:28.005510 31656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:28.005510 31658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:28.005578 31655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:28.006181 31650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:28.006479 31650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:28.007668 31650 hybrid_clock.cc:648] HybridClock initialized: now 1790348368007633 us; error 57 us; skew 500 ppm
Sep 25 14:59:28 dist-test-slave-1sv7 krb5kdc[31637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348368, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:28.010592 31650 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:28.011196 31650 webserver.cc:460] Webserver started at http://127.9.141.126:39339/ using document root <none> and password file <none>
I20260925 14:59:28.011480 31650 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:28.011624 31650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:28.011781 31650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:28.012948 31650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "5bff81c269c24eb1b559f7db5f98e967"
format_stamp: "Formatted at 2026-09-25 14:59:28 on dist-test-slave-1sv7"
server_key: "b05faf291665ce5b6180af5b36c85d20"
server_key_iv: "6b1037e1f1d105e2b66d87b61e42faca"
server_key_version: "encryptionkey@0"
I20260925 14:59:28.013314 31650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "5bff81c269c24eb1b559f7db5f98e967"
format_stamp: "Formatted at 2026-09-25 14:59:28 on dist-test-slave-1sv7"
server_key: "b05faf291665ce5b6180af5b36c85d20"
server_key_iv: "6b1037e1f1d105e2b66d87b61e42faca"
server_key_version: "encryptionkey@0"
I20260925 14:59:28.014799 31650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:28.015913 31665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:28.016271 31650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:28.016431 31650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "5bff81c269c24eb1b559f7db5f98e967"
format_stamp: "Formatted at 2026-09-25 14:59:28 on dist-test-slave-1sv7"
server_key: "b05faf291665ce5b6180af5b36c85d20"
server_key_iv: "6b1037e1f1d105e2b66d87b61e42faca"
server_key_version: "encryptionkey@0"
I20260925 14:59:28.016536 31650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:28.038266 31650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:28.040773 31650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:28.040947 31650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:28.045354 31650 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:35901
I20260925 14:59:28.045439 31717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:35901 every 8 connection(s)
I20260925 14:59:28.045819 31650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:28.047008 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31650
I20260925 14:59:28.047184 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:28.047230 31718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:28.047456 9781 external_mini_cluster.cc:1428] Setting key 9a7585033c4fe4714baa85711ce2770a
I20260925 14:59:28.050369 31718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967: Bootstrap starting.
I20260925 14:59:28.051005 31718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:28.051292 31718 log.cc:824] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:28.051985 31718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967: No bootstrap required, opened a new log
Sep 25 14:59:28 dist-test-slave-1sv7 krb5kdc[31637](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:28.054144 31718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bff81c269c24eb1b559f7db5f98e967" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35901 } }
I20260925 14:59:28.054301 31718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:28.054344 31718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bff81c269c24eb1b559f7db5f98e967, State: Initialized, Role: FOLLOWER
I20260925 14:59:28.054500 31718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bff81c269c24eb1b559f7db5f98e967" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35901 } }
I20260925 14:59:28.054610 31718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:28.054644 31718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:28.054689 31718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:28.055446 31718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bff81c269c24eb1b559f7db5f98e967" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35901 } }
I20260925 14:59:28.055573 31718 leader_election.cc:302] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bff81c269c24eb1b559f7db5f98e967; no voters:
I20260925 14:59:28.055766 31718 leader_election.cc:288] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:28.055846 31723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:28.055997 31723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [term 1 LEADER]: Becoming Leader. State: Replica: 5bff81c269c24eb1b559f7db5f98e967, State: Running, Role: LEADER
I20260925 14:59:28.056032 31718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:28.056080 31723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bff81c269c24eb1b559f7db5f98e967" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35901 } }
I20260925 14:59:28.056439 31724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bff81c269c24eb1b559f7db5f98e967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bff81c269c24eb1b559f7db5f98e967" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35901 } } }
I20260925 14:59:28.056546 31724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:28.056790 31725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bff81c269c24eb1b559f7db5f98e967. Latest consensus state: current_term: 1 leader_uuid: "5bff81c269c24eb1b559f7db5f98e967" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bff81c269c24eb1b559f7db5f98e967" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 35901 } } }
I20260925 14:59:28.056923 31725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:28.057050 31730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:28.057551 31730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:28.058549 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:28.048006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59487 (local address 127.9.141.126:35901)
0925 14:59:28.048232 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:28.048248 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:28.048274 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:59:28.048821 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:28.048830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:28.048882 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:28.049036 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:28.049688 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.050644 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.051531 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.051941 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.052069 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.054487 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:28.054491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:28.054491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:28.054503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:28.056469 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:28.056895 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:28.056898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:28.056899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:28.057001 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:28.057255 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:28.057256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:28.057256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:28.057576 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:28.057727 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:28.058190 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 14:59:28.058331 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260925 14:59:28.059446 31730 catalog_manager.cc:1347] Generated new cluster ID: 1935368cf4e14a04b555b4ddc546301b
I20260925 14:59:28.059530 31730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:28.059617 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:28.067426 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:28.060988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58523 (local address 127.9.141.126:35901)
0925 14:59:28.061042 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:28.061044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:28.061173 (+ 129us) server_negotiation.cc:407] Connection header received
0925 14:59:28.061251 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:28.061252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:28.061262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:28.061319 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:28.061847 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.062333 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.063038 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:28.063276 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:28.063364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:28.064676 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:28.064677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:28.064678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:28.064685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:28.066124 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:28.066510 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:28.066512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:28.066512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:28.066571 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:28.066794 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:28.066794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:28.066795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:28.066935 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:28.067008 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:28.067241 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 14:59:28.067329 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:59:28.069918 31730 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:28.070834 31730 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:28.081566 31730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bff81c269c24eb1b559f7db5f98e967: Generated new TSK 0
I20260925 14:59:28.081816 31730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:28.087853 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31650
2026-09-25T14:59:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:28 dist-test-slave-1sv7 krb5kdc[31749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:28 dist-test-slave-1sv7 krb5kdc[31749](info): set up 2 sockets
Sep 25 14:59:28 dist-test-slave-1sv7 krb5kdc[31749](info): commencing operation
krb5kdc: starting...
W20260925 14:59:29.136713 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:29 dist-test-slave-1sv7 krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:29.261039 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45433
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:45433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:29.342797 31762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:29.343024 31762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:29.343047 31762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:29.344230 31762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:29.344277 31762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:29.344292 31762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:29.344304 31762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:29.344317 31762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:29.345860 31762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:45433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:29.346135 31762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:29.346367 31762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:29.352254 31768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:29.352249 31770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:29.352272 31762 server_base.cc:1034] running on GCE node
W20260925 14:59:29.352234 31767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:29.352864 31762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:29.353183 31762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:29.354374 31762 hybrid_clock.cc:648] HybridClock initialized: now 1790348369354332 us; error 56 us; skew 500 ppm
Sep 25 14:59:29 dist-test-slave-1sv7 krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348369, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:29.357188 31762 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:29.357702 31762 webserver.cc:460] Webserver started at http://127.9.141.126:42925/ using document root <none> and password file <none>
I20260925 14:59:29.357935 31762 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:29.358012 31762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:29.358163 31762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:29.359227 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad"
format_stamp: "Formatted at 2026-09-25 14:59:29 on dist-test-slave-1sv7"
server_key: "f902fcea9c7d67776c28a9652e03434f"
server_key_iv: "b1b5665d3161d22d3b2b973bfd0931b3"
server_key_version: "encryptionkey@0"
I20260925 14:59:29.359594 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad"
format_stamp: "Formatted at 2026-09-25 14:59:29 on dist-test-slave-1sv7"
server_key: "f902fcea9c7d67776c28a9652e03434f"
server_key_iv: "b1b5665d3161d22d3b2b973bfd0931b3"
server_key_version: "encryptionkey@0"
I20260925 14:59:29.361052 31762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:29.361861 31777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:29.362082 31762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:29.362188 31762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad"
format_stamp: "Formatted at 2026-09-25 14:59:29 on dist-test-slave-1sv7"
server_key: "f902fcea9c7d67776c28a9652e03434f"
server_key_iv: "b1b5665d3161d22d3b2b973bfd0931b3"
server_key_version: "encryptionkey@0"
I20260925 14:59:29.362265 31762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:29.380590 31762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:29.382953 31762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:29.383121 31762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:29.387351 31762 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:45433
I20260925 14:59:29.387387 31829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:45433 every 8 connection(s)
I20260925 14:59:29.387734 31762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:29.388793 31830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:29.391023 31830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad: Bootstrap starting.
I20260925 14:59:29.391656 31830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:29.391942 31830 log.cc:824] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:29.392729 31830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad: No bootstrap required, opened a new log
I20260925 14:59:29.394330 31830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45433 } }
I20260925 14:59:29.394464 31830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:29.394487 31830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef5d45a7ec54e20b0b420dc5aa1efad, State: Initialized, Role: FOLLOWER
I20260925 14:59:29.394642 31830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45433 } }
I20260925 14:59:29.394727 31830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:29.394769 31830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:29.394819 31830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:29.395653 31830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45433 } }
I20260925 14:59:29.395735 31830 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef5d45a7ec54e20b0b420dc5aa1efad; no voters:
I20260925 14:59:29.395926 31830 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:29.395985 31833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:29.396139 31833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [term 1 LEADER]: Becoming Leader. State: Replica: 9ef5d45a7ec54e20b0b420dc5aa1efad, State: Running, Role: LEADER
I20260925 14:59:29.396155 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31762
I20260925 14:59:29.396270 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:29.396232 31830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:29.396224 31833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45433 } }
I20260925 14:59:29.396515 9781 external_mini_cluster.cc:1428] Setting key d328d6c0b6574d5d4602834f04296965
I20260925 14:59:29.396585 31833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef5d45a7ec54e20b0b420dc5aa1efad. Latest consensus state: current_term: 1 leader_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45433 } } }
I20260925 14:59:29.396663 31833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:29.396607 31834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef5d45a7ec54e20b0b420dc5aa1efad" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 45433 } } }
I20260925 14:59:29.396752 31834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:29.396874 31838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:29.397424 31838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:29.399160 31838 catalog_manager.cc:1347] Generated new cluster ID: bfb17196eb4a41df94d1e909be1fec6d
I20260925 14:59:29.399214 31838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:29.402212 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:29.397144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53771 (local address 127.9.141.126:45433)
0925 14:59:29.397318 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:29.397329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:29.397605 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:59:29.397742 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:29.397750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:29.397830 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:29.397965 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:29.398294 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.399466 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.400278 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.400753 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.400906 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.401602 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:29.401605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:29.401606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:29.401779 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:29.401811 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:29.401821 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:29.401893 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:29.401983 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:59:29.402048 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260925 14:59:29.403106 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:29.411453 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:29.404645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48623 (local address 127.9.141.126:45433)
0925 14:59:29.404698 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:29.404700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:29.408596 (+ 3896us) server_negotiation.cc:407] Connection header received
0925 14:59:29.408672 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:29.408674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:29.408685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:29.408756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:29.408908 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.409394 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.410165 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:29.410399 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:29.410476 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:29.411159 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:29.411160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:29.411161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:29.411165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:29.411179 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:29.411185 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:29.411242 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:29.411308 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:59:29.411334 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":34}
I20260925 14:59:29.418903 31838 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:29.419456 31838 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:29.423303 31838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef5d45a7ec54e20b0b420dc5aa1efad: Generated new TSK 0
I20260925 14:59:29.423496 31838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:29.426401 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31762
2026-09-25T14:59:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:29 dist-test-slave-1sv7 krb5kdc[31861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:29 dist-test-slave-1sv7 krb5kdc[31861](info): set up 2 sockets
Sep 25 14:59:29 dist-test-slave-1sv7 krb5kdc[31861](info): commencing operation
krb5kdc: starting...
W20260925 14:59:30.468712 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:30 dist-test-slave-1sv7 krb5kdc[31861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:30.597834 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36795
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:40917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:30.680815 31874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:30.681105 31874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:30.681151 31874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:30.682410 31874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:30.682489 31874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:30.682514 31874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:30.682542 31874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:30.682572 31874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:30.684119 31874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:40917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:36795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:30.684449 31874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:30.684711 31874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:30.690011 31880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:30.690021 31882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:30.689971 31879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:30.690397 31874 server_base.cc:1034] running on GCE node
I20260925 14:59:30.690672 31874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:30.690982 31874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:30.692188 31874 hybrid_clock.cc:648] HybridClock initialized: now 1790348370692156 us; error 45 us; skew 500 ppm
Sep 25 14:59:30 dist-test-slave-1sv7 krb5kdc[31861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348370, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:30.694923 31874 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:30.695433 31874 webserver.cc:460] Webserver started at http://127.9.141.126:33867/ using document root <none> and password file <none>
I20260925 14:59:30.695679 31874 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:30.695751 31874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:30.695899 31874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:30.697160 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "e710bf7c48264f41a40529104a5c1b07"
format_stamp: "Formatted at 2026-09-25 14:59:30 on dist-test-slave-1sv7"
server_key: "3073b205ff54e905789052cab8068012"
server_key_iv: "9771d6230e648e923b051d6f9888c1bf"
server_key_version: "encryptionkey@0"
I20260925 14:59:30.697556 31874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "e710bf7c48264f41a40529104a5c1b07"
format_stamp: "Formatted at 2026-09-25 14:59:30 on dist-test-slave-1sv7"
server_key: "3073b205ff54e905789052cab8068012"
server_key_iv: "9771d6230e648e923b051d6f9888c1bf"
server_key_version: "encryptionkey@0"
I20260925 14:59:30.698952 31874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:30.699966 31889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:30.700206 31874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:30.700315 31874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "e710bf7c48264f41a40529104a5c1b07"
format_stamp: "Formatted at 2026-09-25 14:59:30 on dist-test-slave-1sv7"
server_key: "3073b205ff54e905789052cab8068012"
server_key_iv: "9771d6230e648e923b051d6f9888c1bf"
server_key_version: "encryptionkey@0"
I20260925 14:59:30.700415 31874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:30.703816 31874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:30.706156 31874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:30.706312 31874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:30.710462 31874 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:36795
I20260925 14:59:30.710495 31941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:36795 every 8 connection(s)
I20260925 14:59:30.710829 31874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:30.712003 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:30.713254 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31874
I20260925 14:59:30.713369 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:30.713581 9781 external_mini_cluster.cc:1428] Setting key 1a59982fd57ec32f52ba78e0922caa38
I20260925 14:59:30.714771 31942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07: Bootstrap starting.
I20260925 14:59:30.715411 31942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:30.715731 31942 log.cc:824] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:30.716248 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:30.714117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51961 (local address 127.9.141.126:36795)
0925 14:59:30.714304 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:30.714314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:30.714348 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:59:30.714800 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:30.714807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:30.714864 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:30.715038 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:30.715740 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:30.715744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:30.715744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:30.715749 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:30.715793 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:30.715803 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:30.715897 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:30.716055 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:59:30.716098 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260925 14:59:30.716605 31942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07: No bootstrap required, opened a new log
I20260925 14:59:30.717269 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:30.719301 31942 raft_consensus.cc:348] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e710bf7c48264f41a40529104a5c1b07" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36795 } }
I20260925 14:59:30.719497 31942 raft_consensus.cc:374] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:30.719548 31942 raft_consensus.cc:729] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e710bf7c48264f41a40529104a5c1b07, State: Initialized, Role: FOLLOWER
I20260925 14:59:30.719673 31942 consensus_queue.cc:260] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e710bf7c48264f41a40529104a5c1b07" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36795 } }
I20260925 14:59:30.719775 31942 raft_consensus.cc:388] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:30.719830 31942 raft_consensus.cc:482] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:30.719890 31942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:30.719967 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:30.718846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50607 (local address 127.9.141.126:36795)
0925 14:59:30.718898 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:30.718900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:30.718945 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:59:30.719027 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:30.719028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:30.719032 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:30.719078 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:30.719675 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:30.719676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:30.719676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:59:30.719680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:30.719692 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:59:30.719696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:30.719743 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:30.719856 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:59:30.719870 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:59:30.720711 31942 raft_consensus.cc:504] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e710bf7c48264f41a40529104a5c1b07" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36795 } }
I20260925 14:59:30.720804 31942 leader_election.cc:302] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e710bf7c48264f41a40529104a5c1b07; no voters:
I20260925 14:59:30.720950 31942 leader_election.cc:288] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:30.720976 31952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:30.721107 31952 raft_consensus.cc:686] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [term 1 LEADER]: Becoming Leader. State: Replica: e710bf7c48264f41a40529104a5c1b07, State: Running, Role: LEADER
I20260925 14:59:30.721221 31942 sys_catalog.cc:564] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:30.721246 31952 consensus_queue.cc:237] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e710bf7c48264f41a40529104a5c1b07" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36795 } }
I20260925 14:59:30.721683 31953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e710bf7c48264f41a40529104a5c1b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e710bf7c48264f41a40529104a5c1b07" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36795 } } }
I20260925 14:59:30.721729 31954 sys_catalog.cc:455] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e710bf7c48264f41a40529104a5c1b07. Latest consensus state: current_term: 1 leader_uuid: "e710bf7c48264f41a40529104a5c1b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e710bf7c48264f41a40529104a5c1b07" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 36795 } } }
I20260925 14:59:30.721832 31953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:30.721843 31954 sys_catalog.cc:458] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:30.722342 31960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:30.722831 31960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:30.724462 31960 catalog_manager.cc:1347] Generated new cluster ID: 7e52c2b8181b4562a385fa271e42b286
I20260925 14:59:30.724524 31960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:30.751154 31960 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:30.751353 31960 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:30.754266 31960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e710bf7c48264f41a40529104a5c1b07: Generated new TSK 0
I20260925 14:59:30.754467 31960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:59:30.758746 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31874
2026-09-25T14:59:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1332 ms)
[----------] 18 tests from AuthTokenIssuingTest (24209 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:30 dist-test-slave-1sv7 krb5kdc[31973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:30 dist-test-slave-1sv7 krb5kdc[31973](info): set up 2 sockets
Sep 25 14:59:30 dist-test-slave-1sv7 krb5kdc[31973](info): commencing operation
krb5kdc: starting...
W20260925 14:59:31.824664 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:31 dist-test-slave-1sv7 krb5kdc[31973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:31.953536 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37911
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37911,127.9.141.125:36315,127.9.141.124:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:32.034703 31986 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:59:32.034941 31986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:32.034960 31986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:32.034982 31986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:32.036255 31986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:32.036307 31986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:32.036322 31986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:32.036335 31986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:32.036347 31986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:32.038033 31986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37911,127.9.141.125:36315,127.9.141.124:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:37911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:32.038301 31986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:32.038556 31986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:32.043838 31992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:32.043974 31991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:32.043840 31994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:32.044345 31986 server_base.cc:1034] running on GCE node
I20260925 14:59:32.044713 31986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:32.045060 31986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:32.046250 31986 hybrid_clock.cc:648] HybridClock initialized: now 1790348372046236 us; error 36 us; skew 500 ppm
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:32.052946 31986 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:32.053419 31986 webserver.cc:460] Webserver started at http://127.9.141.126:43909/ using document root <none> and password file <none>
I20260925 14:59:32.053642 31986 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:32.053686 31986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:32.053786 31986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:32.054965 31986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "a47ce14d013743a9aebb88ba59fb2a29"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "30e6ef5ffa7b47ff5ca242b3abb4200a"
server_key_iv: "4f2a13738094a61125f110b2d19e5748"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.055307 31986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "a47ce14d013743a9aebb88ba59fb2a29"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "30e6ef5ffa7b47ff5ca242b3abb4200a"
server_key_iv: "4f2a13738094a61125f110b2d19e5748"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.057015 31986 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:59:32.057931 32001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:32.058238 31986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:59:32.058351 31986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "a47ce14d013743a9aebb88ba59fb2a29"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "30e6ef5ffa7b47ff5ca242b3abb4200a"
server_key_iv: "4f2a13738094a61125f110b2d19e5748"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.058442 31986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:32.071671 31986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:32.073982 31986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:32.074146 31986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:32.078109 31986 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:37911
I20260925 14:59:32.078189 32053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:37911 every 8 connection(s)
I20260925 14:59:32.078574 31986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:32.079682 32054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:32.081375 32054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:32.088840 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 31986
I20260925 14:59:32.088973 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:32.089160 9781 external_mini_cluster.cc:1428] Setting key 1accc575d0516dd576886899819e0a20
I20260925 14:59:32.092412 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.082156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.9.141.126:37911)
0925 14:59:32.082346 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.082351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.082416 (+ 65us) server_negotiation.cc:407] Connection header received
0925 14:59:32.082962 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.082964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.082982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.083098 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.083770 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.084903 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.085862 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.086210 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.086310 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.088768 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.088776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.088776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.088782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.090687 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.091131 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.091135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.091136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.091258 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.091540 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.091541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.091541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.091790 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.091856 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.092124 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.092222 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":124,"threads_started":1}
I20260925 14:59:32.092558 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.082120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37911 (local address 127.0.0.1:48612)
0925 14:59:32.082291 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.082333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.082791 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.083137 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.083140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.083460 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.083634 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.083638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.085006 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.085007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.085771 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.085774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.086350 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.086351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.086415 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.087073 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.087082 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.088663 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.090800 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.090807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.090808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.091016 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.091344 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.091345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.091345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.091440 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.091866 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.091873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.091951 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.092088 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.092324 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":67,"thread_start_us":94,"threads_started":1}
I20260925 14:59:32.093334 32054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.093801 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.093653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43026)
0925 14:59:32.093710 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.093769 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 14:59:32.093987 32004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
I20260925 14:59:32.094452 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.094331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43028)
0925 14:59:32.094384 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.094419 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:59:32.094611 32054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:36315: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.141.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab.
Entry for principal kudu/127.9.141.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab.
I20260925 14:59:32.116541 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:36315
--webserver_interface=127.9.141.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab
--principal=kudu/127.9.141.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37911,127.9.141.125:36315,127.9.141.124:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:59:32.143862 32054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } attempt: 1
I20260925 14:59:32.144469 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.144277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43044)
0925 14:59:32.144347 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.144430 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:59:32.144901 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.144801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43050)
0925 14:59:32.144849 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.144880 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:59:32.145005 32054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:36315: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
W20260925 14:59:32.198182 32061 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:59:32.198410 32061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:32.198429 32061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:32.198449 32061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:32.199635 32061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:32.199677 32061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:32.199692 32061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:32.199705 32061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:32.199717 32061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:32.201248 32061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37911,127.9.141.125:36315,127.9.141.124:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab
--principal=kudu/127.9.141.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:36315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.141.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:32.201499 32061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:32.201737 32061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:32.206755 32067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:32.206735 32069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:32.206823 32066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:32.207235 32061 server_base.cc:1034] running on GCE node
I20260925 14:59:32.207489 32061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:32.207795 32061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:32.208959 32061 hybrid_clock.cc:648] HybridClock initialized: now 1790348372208943 us; error 49 us; skew 500 ppm
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:32.211457 32061 init.cc:377] Logged in from keytab as kudu/127.9.141.125@KRBTEST.COM (short username kudu)
I20260925 14:59:32.211930 32061 webserver.cc:460] Webserver started at http://127.9.141.125:43773/ using document root <none> and password file <none>
I20260925 14:59:32.212188 32061 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:32.212260 32061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:32.212468 32061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:32.213141 32054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } attempt: 2
I20260925 14:59:32.213536 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.213385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43058)
0925 14:59:32.213465 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.213504 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:59:32.213781 32061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data/instance:
uuid: "9707673811a845a6af423aebc6950f85"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "9705347f28f819945bd965d28abe83c1"
server_key_iv: "f985c28d7eb7fa748770503fb2dbb99e"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.213950 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.213841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43064)
0925 14:59:32.213884 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.213909 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:59:32.214058 32054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:36315: Network error: Client connection negotiation failed: client connection to 127.9.141.125:36315: connect: Connection refused (error 111)
I20260925 14:59:32.214213 32061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal/instance:
uuid: "9707673811a845a6af423aebc6950f85"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "9705347f28f819945bd965d28abe83c1"
server_key_iv: "f985c28d7eb7fa748770503fb2dbb99e"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.215607 32061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:32.216359 32076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:32.216615 32061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:32.216725 32061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal
uuid: "9707673811a845a6af423aebc6950f85"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "9705347f28f819945bd965d28abe83c1"
server_key_iv: "f985c28d7eb7fa748770503fb2dbb99e"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.216813 32061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:32.241757 32061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:32.243827 32061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:32.244004 32061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:32.248276 32061 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.125:36315
I20260925 14:59:32.248312 32128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.125:36315 every 8 connection(s)
I20260925 14:59:32.248747 32061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
I20260925 14:59:32.249742 32129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:32.251294 32129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.251688 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 32061
I20260925 14:59:32.251786 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-1/wal/instance
I20260925 14:59:32.251937 9781 external_mini_cluster.cc:1428] Setting key bd2f1e5502d233be71f34ff8a094a9eb
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.124@KRBTEST.COM; defaulting to no policy
I20260925 14:59:32.261783 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.252182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.9.141.126:37911)
0925 14:59:32.252248 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.252251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.252486 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:59:32.253001 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.253003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.253019 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.253103 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.253701 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.254499 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.255564 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.255947 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.256028 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.258526 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.258528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.258528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.258537 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.260273 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.260751 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.260752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.260753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.260822 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.261105 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.261105 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.261106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.261219 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.261316 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.261639 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.261715 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:59:32.261946 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.252153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37911 (local address 127.0.0.1:48614)
0925 14:59:32.252308 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.252349 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.252884 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.253157 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.253160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.253401 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.253597 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.253601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.254616 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.254617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.255462 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.255465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.256053 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.256054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.256164 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.256775 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.256784 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.258419 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.260365 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.260371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.260373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.260648 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.260916 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.260917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.260918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.261029 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.261309 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.261315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.261502 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.261590 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.261719 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260925 14:59:32.262749 32129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.124@KRBTEST.COM" created.
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:32.272181 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.263061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43070)
0925 14:59:32.263116 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.263128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.263172 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.263675 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.263676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.263930 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.263992 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.263995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.265565 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.265566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.266422 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.266425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.267078 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.267079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.267138 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.267645 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.267650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.269244 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.271080 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.271081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.271081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.271249 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.271490 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.271491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.271491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.271551 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.271831 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.271832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.271869 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.271942 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.272044 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:59:32.272202 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.263203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.9.141.125:36315)
0925 14:59:32.263443 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.263447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.263465 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:59:32.263548 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.263550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.263565 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.263645 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.264114 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.265428 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.266675 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.266995 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.267061 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.269344 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.269352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.269353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.269361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.270955 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.271340 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.271353 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.271354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.271414 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.271614 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.271615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.271616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.271760 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.271853 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.271968 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.272095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260925 14:59:32.272893 32129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.273286 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.273157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:44974)
0925 14:59:32.273205 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.273257 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:59:32.273418 32079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
I20260925 14:59:32.273947 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.273846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:44986)
0925 14:59:32.273901 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.273925 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:59:32.274097 32129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:37469: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab.
Entry for principal kudu/127.9.141.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab.
I20260925 14:59:32.281737 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:37469
--webserver_interface=127.9.141.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab
--principal=kudu/127.9.141.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:37911,127.9.141.125:36315,127.9.141.124:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:59:32.305208 32054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } attempt: 3
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:32.312542 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.305594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43086)
0925 14:59:32.305662 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.305676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.305752 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.305895 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.305896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.306060 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.306123 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.306125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.306789 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.306790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.307336 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.307338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.307751 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.307752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.307833 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.308319 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.308324 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.309642 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.311419 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.311420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.311420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.311601 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.311803 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.311803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.311804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.311858 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.312186 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.312187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.312231 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.312311 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.312472 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:59:32.312542 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.305687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.9.141.125:36315)
0925 14:59:32.305756 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.305758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.305779 (+ 21us) server_negotiation.cc:407] Connection header received
0925 14:59:32.305809 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.305810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.305821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.305892 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.306208 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.306704 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.307429 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.307653 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.307750 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.309738 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.309739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.309740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.309747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.311312 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.311674 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.311675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.311676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.311735 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.311960 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.311961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.311961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.312096 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.312165 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.312349 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.312450 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:59:32.313123 32054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.313479 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.313363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45000)
0925 14:59:32.313419 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.313454 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:59:32.313943 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.313841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45010)
0925 14:59:32.313894 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.313922 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:59:32.314043 32054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:37469: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
I20260925 14:59:32.323365 32129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } attempt: 1
I20260925 14:59:32.323841 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.323726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45016)
0925 14:59:32.323770 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.323811 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:59:32.324270 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.324176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45026)
0925 14:59:32.324223 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.324250 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:59:32.324422 32129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:37469: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
I20260925 14:59:32.345208 32054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } attempt: 1
I20260925 14:59:32.345772 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.345611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45032)
0925 14:59:32.345696 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.345736 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:59:32.346220 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.346123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45034)
0925 14:59:32.346171 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.346198 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:59:32.346338 32054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:37469: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
W20260925 14:59:32.365777 32136 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:59:32.366061 32136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:32.366084 32136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:32.366106 32136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:32.367188 32136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:32.367221 32136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:32.367235 32136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:32.367247 32136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:32.367259 32136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:32.368697 32136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:37911,127.9.141.125:36315,127.9.141.124:37469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab
--principal=kudu/127.9.141.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.141.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:32.368889 32136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:32.369063 32136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:32.373975 32141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:32.374020 32144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:32.373975 32142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:32.374344 32136 server_base.cc:1034] running on GCE node
I20260925 14:59:32.374627 32136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:32.374871 32136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:32.375998 32136 hybrid_clock.cc:648] HybridClock initialized: now 1790348372375993 us; error 36 us; skew 500 ppm
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:32.378484 32136 init.cc:377] Logged in from keytab as kudu/127.9.141.124@KRBTEST.COM (short username kudu)
I20260925 14:59:32.378983 32136 webserver.cc:460] Webserver started at http://127.9.141.124:40363/ using document root <none> and password file <none>
I20260925 14:59:32.379206 32136 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:32.379272 32136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:32.379429 32136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:32.380543 32136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data/instance:
uuid: "0ebe4e8417694deeb1fa67601a616ee2"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "aabe683f9c2116f9d956deba21be25ad"
server_key_iv: "560ead00caa1edcc8c4495e49cf098ba"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.380894 32136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal/instance:
uuid: "0ebe4e8417694deeb1fa67601a616ee2"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "aabe683f9c2116f9d956deba21be25ad"
server_key_iv: "560ead00caa1edcc8c4495e49cf098ba"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.382257 32136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:59:32.383167 32151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:32.383396 32136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:32.383529 32136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal
uuid: "0ebe4e8417694deeb1fa67601a616ee2"
format_stamp: "Formatted at 2026-09-25 14:59:32 on dist-test-slave-1sv7"
server_key: "aabe683f9c2116f9d956deba21be25ad"
server_key_iv: "560ead00caa1edcc8c4495e49cf098ba"
server_key_version: "encryptionkey@0"
I20260925 14:59:32.383606 32136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:32.392553 32129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } attempt: 2
I20260925 14:59:32.393044 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.392891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45044)
0925 14:59:32.392972 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.393012 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 14:59:32.393397 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.393318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45048)
0925 14:59:32.393359 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.393382 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:59:32.393504 32129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:37469: Network error: Client connection negotiation failed: client connection to 127.9.141.124:37469: connect: Connection refused (error 111)
I20260925 14:59:32.396014 32136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:32.398234 32136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:32.398340 32136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:32.402606 32136 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.124:37469
I20260925 14:59:32.402637 32203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.124:37469 every 8 connection(s)
I20260925 14:59:32.403052 32136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
I20260925 14:59:32.404002 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:32.405534 32204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.406713 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 32136
I20260925 14:59:32.406781 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348105363960-9781-0/minicluster-data/master-2/wal/instance
I20260925 14:59:32.407068 9781 external_mini_cluster.cc:1428] Setting key 80944215b60b3cd3f37cf4900b940f87
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:59:32.415225 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.406188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.9.141.126:37911)
0925 14:59:32.406298 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.406300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.406486 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:59:32.407018 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.407020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.407033 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.407090 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.407807 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.408288 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.409298 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.409512 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.409599 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.411711 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.411712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.411712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.411720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.413634 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.414142 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.414143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.414144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.414216 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.414501 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.414502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.414502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.414670 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.414737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.415090 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.415158 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:59:32.415469 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.406170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:37911 (local address 127.0.0.1:48626)
0925 14:59:32.406333 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.406374 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.406869 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.407161 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.407166 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.407483 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.407714 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.407719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.408414 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.408415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.409195 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.409197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.409626 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.409628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.409741 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.410260 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.410268 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.411621 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.413708 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.413712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.413714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.414043 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.414285 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.414286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.414286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.414430 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.414752 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.414759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.414928 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.415057 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.415201 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260925 14:59:32.416247 32204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.417146 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.407607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.9.141.124:37469)
0925 14:59:32.407769 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.407775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.407906 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:59:32.407973 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.407976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.408007 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.408088 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.408754 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.409472 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.410385 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.410678 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.410768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.413659 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.413666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.413667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.413693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.415583 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.416035 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.416038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.416039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.416112 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.416357 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.416358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.416358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.416637 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.416779 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.416954 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.417030 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:32.421491 32054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } attempt: 2
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:32.424652 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.416552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:36315 (local address 127.0.0.1:43100)
0925 14:59:32.416591 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.416600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.416631 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.416826 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.416827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.416968 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.417016 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.417018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.417952 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.417953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.418503 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.418505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.419107 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.419108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.419168 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.419731 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.419735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.421341 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.423356 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.423357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.423357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.423628 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.423860 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.423861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.423861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.423925 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.424301 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.424302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.424346 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.424467 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.424555 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":2}
I20260925 14:59:32.424952 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.416637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.9.141.125:36315)
0925 14:59:32.416708 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.416710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.416717 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:59:32.416741 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.416742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.416759 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.416823 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.417112 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.417853 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.418765 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.418987 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.419077 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.421433 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.421435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.421435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.421451 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.423245 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.423719 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.423720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.423720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.423780 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.424052 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.424054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.424054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.424207 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.424293 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.424519 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.424597 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:59:32.425537 32204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:32.425935 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.417849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.9.141.126:37911)
0925 14:59:32.417883 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.417885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.417893 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:32.417956 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.417957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.417966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.418018 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.418499 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.418988 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.419951 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.420323 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.420445 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.422991 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.422992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.422992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.422998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.424739 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.425147 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.425149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.425149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.425200 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.425376 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.425377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.425377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.425484 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.425711 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.425753 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.425845 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:59:32.429734 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.421857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45068)
0925 14:59:32.421906 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.421916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.421968 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.422147 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.422148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.422296 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.422354 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.422356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.423912 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.423913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.424677 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.424680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.425057 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.425058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.425113 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.425567 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.425572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.426991 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.428672 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.428672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.428673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.428883 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.429095 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.429095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.429096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.429161 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.429439 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.429441 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.429482 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.429541 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.429625 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:59:32.430022 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.421932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.9.141.124:37469)
0925 14:59:32.421973 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.421975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.422019 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:59:32.422048 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.422049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.422057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.422121 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.422453 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.423822 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.424770 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.424983 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.425058 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.427074 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.427075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.427075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.427082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.428596 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.428958 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.428959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.428959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.429017 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.429232 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.429233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.429234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.429352 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.429411 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.429865 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.429937 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:32.432786 32054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29: Bootstrap starting.
I20260925 14:59:32.433522 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.425930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45082)
0925 14:59:32.426012 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.426022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.426059 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.426419 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.426419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.426570 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.426599 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.426601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.427307 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.427307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.428305 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.428308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.428810 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.428812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.428874 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.429264 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.429269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.430422 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.432328 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.432329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.432329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.432568 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.432790 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.432791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.432791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.432846 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.433192 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.433193 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.433243 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.433314 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.433393 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 14:59:32.433555 32054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:32.433833 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.426077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.9.141.124:37469)
0925 14:59:32.426275 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.426276 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.426283 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:59:32.426302 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.426302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.426311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.426375 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.426663 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.427210 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.428427 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.428728 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.428803 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.430488 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.430489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.430489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.430496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.432244 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.432650 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.432651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.432652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.432717 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.432918 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.432919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.432919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.433109 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.433181 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.433635 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.433715 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":125,"threads_started":1}
I20260925 14:59:32.433866 32054 log.cc:824] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:32.434657 32054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29: No bootstrap required, opened a new log
I20260925 14:59:32.435544 32204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: Bootstrap starting.
I20260925 14:59:32.436210 32204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:32.436539 32204 log.cc:824] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:32.436880 32054 raft_consensus.cc:348] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.437040 32054 raft_consensus.cc:729] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47ce14d013743a9aebb88ba59fb2a29, State: Initialized, Role: FOLLOWER
I20260925 14:59:32.437129 32054 consensus_queue.cc:260] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.437296 32204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: No bootstrap required, opened a new log
I20260925 14:59:32.437506 32054 sys_catalog.cc:564] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:32.437490 32212 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.437709 32212 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:32.438431 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.426771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.9.141.125:36315)
0925 14:59:32.426806 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.426808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.427406 (+ 598us) server_negotiation.cc:407] Connection header received
0925 14:59:32.427443 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.427444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.427454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.427505 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.428085 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.428733 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.429706 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.430382 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.430429 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.433067 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.433069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.433069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.433075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.437129 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.437599 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.437600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.437601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.437662 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.437851 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.437852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.437852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.438006 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.438208 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.438253 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.438339 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:59:32.439489 32223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:32.439560 32223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:32.439612 32204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.439761 32204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebe4e8417694deeb1fa67601a616ee2, State: Initialized, Role: FOLLOWER
I20260925 14:59:32.439844 32204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.440129 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:32.440245 32204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:32.440522 32225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.440601 32225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:59:32.442196 32236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:32.442238 32236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:32.440928 31986 master_runner.cc:186] Error getting master registration for 127.9.141.125:36315: OK, instance_id { permanent_uuid: "9707673811a845a6af423aebc6950f85" instance_seqno: 1790348372247109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:59:32.443655 31986 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:59:32.444161 32028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a47ce14d013743a9aebb88ba59fb2a29"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48638
I20260925 14:59:32.444322 32028 raft_consensus.cc:482] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:59:32.444415 32028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:32.445358 32028 raft_consensus.cc:504] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.445609 32028 leader_election.cc:288] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [CANDIDATE]: Term 1 election: Requested vote from peers 9707673811a845a6af423aebc6950f85 (127.9.141.125:36315), 0ebe4e8417694deeb1fa67601a616ee2 (127.9.141.124:37469)
I20260925 14:59:32.446125 32178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47ce14d013743a9aebb88ba59fb2a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ebe4e8417694deeb1fa67601a616ee2"
I20260925 14:59:32.446228 32178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:32.447154 32178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47ce14d013743a9aebb88ba59fb2a29 in term 1.
I20260925 14:59:32.447314 32004 leader_election.cc:302] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebe4e8417694deeb1fa67601a616ee2, a47ce14d013743a9aebb88ba59fb2a29; no voters:
I20260925 14:59:32.447425 32212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:32.447583 32212 raft_consensus.cc:686] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [term 1 LEADER]: Becoming Leader. State: Replica: a47ce14d013743a9aebb88ba59fb2a29, State: Running, Role: LEADER
I20260925 14:59:32.447671 32212 consensus_queue.cc:237] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.448053 32212 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47ce14d013743a9aebb88ba59fb2a29. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.448140 32212 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:32.448053 32136 master_runner.cc:186] Error getting master registration for 127.9.141.125:36315: OK, instance_id { permanent_uuid: "9707673811a845a6af423aebc6950f85" instance_seqno: 1790348372247109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:59:32.451126 32136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:59:32.452605 32246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:32.453141 32246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:32.456099 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.447369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.9.141.125:36315)
0925 14:59:32.447414 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.447417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.448191 (+ 774us) server_negotiation.cc:407] Connection header received
0925 14:59:32.448287 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.448288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.448298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.448328 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.449145 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.449871 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.450918 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.451230 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.451273 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.452927 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.452928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.452929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.452935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.454607 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.454986 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.454987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.454987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.455038 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.455433 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.455434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.455435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.455598 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.455679 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.455913 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.455998 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:59:32.461719 32178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [term 1 FOLLOWER]: Refusing update from remote peer a47ce14d013743a9aebb88ba59fb2a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:32.462071 32212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:32.463538 32225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47ce14d013743a9aebb88ba59fb2a29. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.463639 32225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:32.464761 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.447847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.9.141.124:37469)
0925 14:59:32.447900 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.447902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.455902 (+ 8000us) server_negotiation.cc:407] Connection header received
0925 14:59:32.456059 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.456060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.456072 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.456115 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.457361 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.458133 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.459149 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.459486 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.459526 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.461128 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.461130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.461130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.461137 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.463310 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.463815 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.463816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.463817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.463876 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.464126 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.464127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.464127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.464262 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.464368 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.464585 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.464671 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:59:32.466841 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.455527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.9.141.126:37911)
0925 14:59:32.455569 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.455571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.456890 (+ 1319us) server_negotiation.cc:407] Connection header received
0925 14:59:32.456990 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.456991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.457003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.457042 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.458162 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.458968 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.460693 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.461018 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.461062 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.463214 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.463216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.463216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.463224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.465201 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.465858 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.465859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.465860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.465947 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.466156 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.466157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.466157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.466283 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.466389 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.466662 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.466752 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:59:32.469709 32212 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.469812 32212 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:32.470326 32246 catalog_manager.cc:1347] Generated new cluster ID: 730963b59fef48fdbe52bfb50e8e6f95
I20260925 14:59:32.470376 32246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:32.470331 32225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.470407 32225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:32.484647 32129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } attempt: 3
Sep 25 14:59:32 dist-test-slave-1sv7 krb5kdc[31973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348372, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:59:32.495095 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.484959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:37469 (local address 127.0.0.1:45104)
0925 14:59:32.485065 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:32.485079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:32.485131 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:32.485321 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:32.485322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:32.485571 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:32.485624 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.485626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.486796 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.486797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.487730 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:32.487734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.489346 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.489348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:32.489430 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.490072 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:32.490078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:32.491682 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:32.493611 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.493613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.493613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.493905 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.494229 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:32.494230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:32.494230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:32.494311 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:32.494701 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:32.494702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:32.494769 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 14:59:32.494840 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:59:32.494960 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 14:59:32.496838 32209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:32.485081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.9.141.124:37469)
0925 14:59:32.485129 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.485131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.485141 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:32.485207 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.485208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.485219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.485270 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.485730 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.486578 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.488807 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:32.489186 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:32.489261 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:32.491789 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:32.491791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:32.491791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:32.491800 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:32.493515 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.494050 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.494051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.494052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.494136 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:32.494397 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:32.494398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:32.494398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:32.494585 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:32.494652 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:32.496630 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0925 14:59:32.496734 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:59:32.498962 32129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: Bootstrap starting.
I20260925 14:59:32.499748 32129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:32.500069 32129 log.cc:824] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:32.502198 32246 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:32.503017 32246 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:32.505129 32129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: No bootstrap required, opened a new log
I20260925 14:59:32.507247 32129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.507395 32129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9707673811a845a6af423aebc6950f85, State: Initialized, Role: FOLLOWER
I20260925 14:59:32.507473 32129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } }
I20260925 14:59:32.507822 32129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:32.508896 32102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:32.509857 32102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [term 1 FOLLOWER]: Refusing update from remote peer a47ce14d013743a9aebb88ba59fb2a29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:32.510247 32212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:32.510565 32103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a47ce14d013743a9aebb88ba59fb2a29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9707673811a845a6af423aebc6950f85"
I20260925 14:59:32.511375 32103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47ce14d013743a9aebb88ba59fb2a29 in term 1.
I20260925 14:59:32.513762 32258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.513882 32258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:32.513947 32258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a47ce14d013743a9aebb88ba59fb2a29. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.514005 32258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:32.514037 32258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.514081 32258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:32.514499 32246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29: Generated new TSK 0
I20260925 14:59:32.514727 32246 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:59:32.515497 32269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:32.515601 32269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:32.517038 32243 sys_catalog.cc:455] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a47ce14d013743a9aebb88ba59fb2a29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47ce14d013743a9aebb88ba59fb2a29" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 37911 } } peers { permanent_uuid: "9707673811a845a6af423aebc6950f85" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 36315 } } peers { permanent_uuid: "0ebe4e8417694deeb1fa67601a616ee2" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 37469 } } }
I20260925 14:59:32.517130 32243 sys_catalog.cc:458] T 00000000000000000000000000000000 P a47ce14d013743a9aebb88ba59fb2a29 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:32.520918 32261 mvcc.cc:205] Tried to move back new op lower bound from 7333266933764358144 to 7333266933546307584. Current Snapshot: MvccSnapshot[applied={T|T < 7333266933599989760}]
W20260925 14:59:32.537421 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.535776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.9.141.125:36315)
0925 14:59:32.535821 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.535823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.536670 (+ 847us) server_negotiation.cc:407] Connection header received
0925 14:59:32.536793 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.536795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.536807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.536841 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.537339 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43122: BlockingRecv error: recv got EOF from 127.0.0.1:43122 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:59:32.538517 32209 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.536183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.9.141.124:37469)
0925 14:59:32.536240 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.536242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.537933 (+ 1691us) server_negotiation.cc:407] Connection header received
0925 14:59:32.538054 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.538056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.538069 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.538101 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.538432 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45114: BlockingRecv error: recv got EOF from 127.0.0.1:45114 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:59:32.539548 32057 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.535638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.9.141.126:37911)
0925 14:59:32.535692 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.535695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.537685 (+ 1990us) server_negotiation.cc:407] Connection header received
0925 14:59:32.537791 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.537792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.537804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.537911 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.539476 (+ 1565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48654: BlockingRecv error: recv got EOF from 127.0.0.1:48654 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 14:59:32.541503 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.540732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.9.141.125:36315)
0925 14:59:32.540771 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.540773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.540980 (+ 207us) server_negotiation.cc:407] Connection header received
0925 14:59:32.541086 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.541087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.541106 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.541136 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.541456 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43136: BlockingRecv error: recv got EOF from 127.0.0.1:43136 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:59:32.544658 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.543844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.9.141.125:36315)
0925 14:59:32.543877 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.543879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.544120 (+ 241us) server_negotiation.cc:407] Connection header received
0925 14:59:32.544215 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.544216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.544226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.544255 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.544610 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43142: BlockingRecv error: recv got EOF from 127.0.0.1:43142 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:59:32.548545 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.547746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43152 (local address 127.9.141.125:36315)
0925 14:59:32.547780 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.547782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.548016 (+ 234us) server_negotiation.cc:407] Connection header received
0925 14:59:32.548106 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.548107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.548117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.548146 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.548492 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43152: BlockingRecv error: recv got EOF from 127.0.0.1:43152 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:59:32.554519 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.553670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.9.141.125:36315)
0925 14:59:32.553707 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.553709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.553965 (+ 256us) server_negotiation.cc:407] Connection header received
0925 14:59:32.554068 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.554070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.554081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.554113 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.554467 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43154: BlockingRecv error: recv got EOF from 127.0.0.1:43154 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:59:32.564775 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.563793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.9.141.125:36315)
0925 14:59:32.563837 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.563839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.564127 (+ 288us) server_negotiation.cc:407] Connection header received
0925 14:59:32.564236 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.564237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.564249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.564282 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.564718 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43164: BlockingRecv error: recv got EOF from 127.0.0.1:43164 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:59:32.583339 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.582156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.9.141.125:36315)
0925 14:59:32.582200 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.582203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.582526 (+ 323us) server_negotiation.cc:407] Connection header received
0925 14:59:32.582640 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.582642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.582655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.582687 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.583258 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43166: BlockingRecv error: recv got EOF from 127.0.0.1:43166 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:59:32.617707 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.616757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.9.141.125:36315)
0925 14:59:32.616808 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.616811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.617095 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:59:32.617196 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.617197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.617209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.617239 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.617643 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43174: BlockingRecv error: recv got EOF from 127.0.0.1:43174 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 14:59:32.683871 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.683148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.9.141.125:36315)
0925 14:59:32.683211 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.683213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.683246 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:59:32.683345 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.683346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.683357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.683430 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.683786 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43186: BlockingRecv error: recv got EOF from 127.0.0.1:43186 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 14:59:32.814103 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:32.813223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.9.141.125:36315)
0925 14:59:32.813265 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:32.813269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:32.813418 (+ 149us) server_negotiation.cc:407] Connection header received
0925 14:59:32.813514 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:32.813515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:32.813528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:32.813605 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:32.814033 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43192: BlockingRecv error: recv got EOF from 127.0.0.1:43192 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:59:33.072278 32133 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:33.071452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.9.141.125:36315)
0925 14:59:33.071538 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.071541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.071652 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:59:33.071694 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.071695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.071707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.071766 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:33.072216 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43208: BlockingRecv error: recv got EOF from 127.0.0.1:43208 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:59:33.442695 32236 catalog_manager.cc:1259] Loaded cluster ID: 730963b59fef48fdbe52bfb50e8e6f95
I20260925 14:59:33.442765 32236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: loading cluster ID for follower catalog manager: success
I20260925 14:59:33.443677 32236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: acquiring CA information for follower catalog manager: success
I20260925 14:59:33.444072 32236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ebe4e8417694deeb1fa67601a616ee2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:59:33.516139 32269 catalog_manager.cc:1259] Loaded cluster ID: 730963b59fef48fdbe52bfb50e8e6f95
I20260925 14:59:33.516220 32269 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: loading cluster ID for follower catalog manager: success
I20260925 14:59:33.517416 32269 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: acquiring CA information for follower catalog manager: success
I20260925 14:59:33.517834 32269 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9707673811a845a6af423aebc6950f85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:59:33.588657 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.585579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.9.141.125:36315)
0925 14:59:33.585719 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.585721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.585804 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:59:33.585885 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.585886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.585943 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.586023 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:33.586269 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.586770 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.587764 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.587976 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.588037 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.588225 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:33.588333 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:33.588514 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:33.588569 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:59:33.588590 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260925 14:59:33.593358 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.590801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.9.141.124:37469)
0925 14:59:33.590955 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.590957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.590964 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:59:33.591000 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.591000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.591034 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.591103 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:33.591169 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.591671 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.592503 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.592721 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.592796 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.592970 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:33.593080 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:33.593149 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:33.593187 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:59:33.593215 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260925 14:59:33.598294 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.595343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.9.141.125:36315)
0925 14:59:33.595385 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.595387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.595491 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:59:33.595589 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.595591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.595602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.595655 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:33.595844 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.596635 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.597445 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.597751 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.597811 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.597996 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:33.598034 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:33.598082 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:33.598194 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:59:33.598218 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:59:33.598613 32345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.595618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.9.141.124:37469)
0925 14:59:33.595656 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.595657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.595668 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:59:33.595704 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.595704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.595715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.595765 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:33.595952 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.596496 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.597578 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.597868 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.597922 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.598089 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:33.598122 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:33.598175 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:33.598500 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 14:59:33.598540 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:33.598742 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:33.595310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.9.141.126:37911)
0925 14:59:33.595649 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:33.595652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:33.595660 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:33.595706 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:33.595707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:33.595790 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:33.595873 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:33.595991 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.596777 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.597771 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:33.598023 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:33.598088 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:33.598302 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:33.598413 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:33.598476 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:33.598629 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:59:33.598653 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":66,"threads_started":1}
I20260925 14:59:33.600265 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 31986
I20260925 14:59:33.605792 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 32061
I20260925 14:59:33.611122 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 32136
2026-09-25T14:59:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2854 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:33 dist-test-slave-1sv7 krb5kdc[32356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:33 dist-test-slave-1sv7 krb5kdc[32356](info): set up 2 sockets
Sep 25 14:59:33 dist-test-slave-1sv7 krb5kdc[32356](info): commencing operation
krb5kdc: starting...
W20260925 14:59:34.684801 9781 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/test-user.keytab.
Sep 25 14:59:34 dist-test-slave-1sv7 krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.141.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
Entry for principal kudu/127.9.141.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab.
I20260925 14:59:34.813712 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:33651
--webserver_interface=127.9.141.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:33651,127.9.141.125:43617,127.9.141.124:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:34.896461 32369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:59:34.896687 32369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:34.896703 32369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:34.896725 32369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:34.897837 32369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:34.897877 32369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:34.897891 32369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:34.897904 32369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:34.897917 32369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:34.899461 32369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:33651,127.9.141.125:43617,127.9.141.124:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.126.keytab
--principal=kudu/127.9.141.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.126:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.141.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:34.899668 32369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:34.899842 32369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:34.905040 32375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:34.905014 32377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:34.905061 32374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:34.905261 32369 server_base.cc:1034] running on GCE node
I20260925 14:59:34.905546 32369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:34.905804 32369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:34.906960 32369 hybrid_clock.cc:648] HybridClock initialized: now 1790348374906942 us; error 30 us; skew 500 ppm
Sep 25 14:59:34 dist-test-slave-1sv7 krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:34.909650 32369 init.cc:377] Logged in from keytab as kudu/127.9.141.126@KRBTEST.COM (short username kudu)
I20260925 14:59:34.910125 32369 webserver.cc:460] Webserver started at http://127.9.141.126:44343/ using document root <none> and password file <none>
I20260925 14:59:34.910359 32369 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:34.910410 32369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:34.910563 32369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:34.911725 32369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data/instance:
uuid: "90e40cb46cec4ec988f5689c0e28b478"
format_stamp: "Formatted at 2026-09-25 14:59:34 on dist-test-slave-1sv7"
server_key: "b68efb566e7c05badf13b33bab36c230"
server_key_iv: "2dea8f765d4be0e9cbf076353ddb8900"
server_key_version: "encryptionkey@0"
I20260925 14:59:34.912114 32369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal/instance:
uuid: "90e40cb46cec4ec988f5689c0e28b478"
format_stamp: "Formatted at 2026-09-25 14:59:34 on dist-test-slave-1sv7"
server_key: "b68efb566e7c05badf13b33bab36c230"
server_key_iv: "2dea8f765d4be0e9cbf076353ddb8900"
server_key_version: "encryptionkey@0"
I20260925 14:59:34.913645 32369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:34.914537 32384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:34.914742 32369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:34.914875 32369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal
uuid: "90e40cb46cec4ec988f5689c0e28b478"
format_stamp: "Formatted at 2026-09-25 14:59:34 on dist-test-slave-1sv7"
server_key: "b68efb566e7c05badf13b33bab36c230"
server_key_iv: "2dea8f765d4be0e9cbf076353ddb8900"
server_key_version: "encryptionkey@0"
I20260925 14:59:34.914973 32369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:34.928552 32369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:34.930799 32369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:34.930964 32369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:34.935230 32369 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.126:33651
I20260925 14:59:34.935264 32436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.126:33651 every 8 connection(s)
I20260925 14:59:34.935631 32369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:34.936715 32437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:34.938341 32437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:34.939545 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 32369
I20260925 14:59:34.939639 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-0/wal/instance
I20260925 14:59:34.939845 9781 external_mini_cluster.cc:1428] Setting key 9ca4d17c44562f90f5399911811ce81a
Sep 25 14:59:34 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.125@KRBTEST.COM; defaulting to no policy
I20260925 14:59:34.949977 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.939055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:33651 (local address 127.0.0.1:46958)
0925 14:59:34.939320 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:34.939361 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:34.939980 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:34.940320 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:34.940323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:34.940727 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:34.940907 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:34.940912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.942052 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.942053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:34.943218 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:34.943221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.943817 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.943819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:34.943907 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.944579 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:34.944587 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:34.946214 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:34.948447 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:34.948455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:34.948458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:34.948663 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:34.948901 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:34.948902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:34.948902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:34.948987 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:34.949492 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:34.949498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:34.949560 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 14:59:34.949634 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:59:34.949757 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":197,"threads_started":1}
I20260925 14:59:34.949981 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.939123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.9.141.126:33651)
0925 14:59:34.939302 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:34.939311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:34.939517 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:59:34.940134 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:34.940136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:34.940159 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:34.940273 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:34.941036 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.941948 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.943311 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:34.943696 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:34.943779 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:34.946309 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:34.946319 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:34.946319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:34.946327 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:34.948300 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:34.948749 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:34.948752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:34.948753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:34.948833 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:34.949128 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:34.949129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:34.949130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:34.949411 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:34.949483 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:34.949729 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:59:34.949815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260925 14:59:34.950942 32437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.125@KRBTEST.COM" created.
I20260925 14:59:34.951502 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.951194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34566)
0925 14:59:34.951352 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:34.951458 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260925 14:59:34.951887 32387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
I20260925 14:59:34.952463 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:34.952266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34572)
0925 14:59:34.952323 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:34.952359 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:59:34.952591 32437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:43617: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab.
Entry for principal kudu/127.9.141.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab.
I20260925 14:59:34.968973 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:43617
--webserver_interface=127.9.141.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab
--principal=kudu/127.9.141.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:33651,127.9.141.125:43617,127.9.141.124:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:59:35.001901 32437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } attempt: 1
I20260925 14:59:35.002532 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.002382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34576)
0925 14:59:35.002451 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.002498 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:59:35.003062 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.002963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34582)
0925 14:59:35.003014 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.003042 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:59:35.003225 32437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:43617: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
W20260925 14:59:35.050274 32444 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:59:35.050570 32444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:35.050601 32444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:35.050639 32444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:35.051805 32444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:35.051882 32444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:35.051908 32444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:35.051935 32444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:35.051965 32444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:35.053503 32444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:33651,127.9.141.125:43617,127.9.141.124:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.125.keytab
--principal=kudu/127.9.141.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.125:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.141.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:35.053781 32444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:35.054019 32444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:35.059024 32450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:35.059057 32444 server_base.cc:1034] running on GCE node
W20260925 14:59:35.059029 32449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:35.059041 32452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:35.059499 32444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:35.059782 32444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:35.060941 32444 hybrid_clock.cc:648] HybridClock initialized: now 1790348375060929 us; error 30 us; skew 500 ppm
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:35.063508 32444 init.cc:377] Logged in from keytab as kudu/127.9.141.125@KRBTEST.COM (short username kudu)
I20260925 14:59:35.064029 32444 webserver.cc:460] Webserver started at http://127.9.141.125:35027/ using document root <none> and password file <none>
I20260925 14:59:35.064289 32444 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:35.064363 32444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:35.064544 32444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:35.065624 32444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data/instance:
uuid: "a5256b21189e48a5a01c0f2a4037db86"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-1sv7"
server_key: "617ddc559ba8523e8d415f746055b193"
server_key_iv: "ae1f363fc72287d9c3805d223326cea3"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.065987 32444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal/instance:
uuid: "a5256b21189e48a5a01c0f2a4037db86"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-1sv7"
server_key: "617ddc559ba8523e8d415f746055b193"
server_key_iv: "ae1f363fc72287d9c3805d223326cea3"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.067993 32444 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:35.068984 32459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:35.069186 32444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:59:35.069298 32444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal
uuid: "a5256b21189e48a5a01c0f2a4037db86"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-1sv7"
server_key: "617ddc559ba8523e8d415f746055b193"
server_key_iv: "ae1f363fc72287d9c3805d223326cea3"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.069387 32444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:35.071411 32437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } attempt: 2
I20260925 14:59:35.071861 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.071709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34586)
0925 14:59:35.071780 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.071826 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:59:35.072234 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.072140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34602)
0925 14:59:35.072188 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.072214 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:59:35.072395 32437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.125:43617: Network error: Client connection negotiation failed: client connection to 127.9.141.125:43617: connect: Connection refused (error 111)
I20260925 14:59:35.082355 32444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:35.084599 32444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:35.084764 32444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:35.088969 32444 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.125:43617
I20260925 14:59:35.089133 32511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.125:43617 every 8 connection(s)
I20260925 14:59:35.089412 32444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/data/info.pb
I20260925 14:59:35.090422 32512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:35.091925 32512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:35.094099 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 32444
I20260925 14:59:35.094203 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-1/wal/instance
I20260925 14:59:35.094472 9781 external_mini_cluster.cc:1428] Setting key 4b57f67fb1827814a76b755e4a7f9bb9
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.141.124@KRBTEST.COM; defaulting to no policy
I20260925 14:59:35.102491 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.092775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.9.141.126:33651)
0925 14:59:35.092847 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.092849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.093015 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:59:35.093547 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.093548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.093563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.093610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.094501 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.095024 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.096402 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.096738 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.096822 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.099129 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.099130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.099131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.099139 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.100897 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.101439 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.101440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.101441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.101508 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.101758 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.101759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.101760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.101924 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.102000 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.102270 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.102368 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:59:35.102636 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.092630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:33651 (local address 127.0.0.1:46964)
0925 14:59:35.092880 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.092925 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.093428 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.093678 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.093682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.094068 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.094373 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.094379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.095126 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.095127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.096276 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.096278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.096837 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.096859 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.096973 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.097577 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.097586 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.099023 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.101021 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.101030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.101032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.101312 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.101585 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.101585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.101585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.101684 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.102005 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.102012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.102132 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.102221 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.102406 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260925 14:59:35.103250 32512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.141.124@KRBTEST.COM" created.
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:35.111299 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.103717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.9.141.125:43617)
0925 14:59:35.103873 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.103877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.103893 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:59:35.103909 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.103911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.103921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.104012 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.104282 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.105074 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.105873 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.106178 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.106269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.108266 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.108273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.108274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.108279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.109919 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.110445 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.110448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.110449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.110508 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.110737 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.110737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.110738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.110895 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.110961 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.111139 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.111212 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260925 14:59:35.111440 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.103571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34610)
0925 14:59:35.103641 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.103653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.103703 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.104011 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.104012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.104150 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.104205 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.104206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.105191 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.105192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.105777 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.105779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.106267 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.106268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.106332 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.106771 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.106777 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.108178 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.110040 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.110041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.110042 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.110284 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.110589 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.110590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.110590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.110665 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.110969 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.110971 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.111026 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.111117 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.111292 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":53}
I20260925 14:59:35.112084 32512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:35.112476 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.112320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54284)
0925 14:59:35.112371 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.112446 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:59:35.112658 32463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
I20260925 14:59:35.113128 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.113014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54288)
0925 14:59:35.113065 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.113097 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:59:35.113261 32512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:45989: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.141.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab.
Entry for principal kudu/127.9.141.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab.
I20260925 14:59:35.124102 9781 external_mini_cluster.cc:1326] Running /tmp/dist-test-task5afOW1/build/release/bin/kudu
/tmp/dist-test-task5afOW1/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:45989
--webserver_interface=127.9.141.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab
--principal=kudu/127.9.141.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.141.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.141.126:33651,127.9.141.125:43617,127.9.141.124:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:59:35.162559 32512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } attempt: 1
I20260925 14:59:35.163228 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.163053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54304)
0925 14:59:35.163150 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.163195 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 14:59:35.163548 32437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } attempt: 3
I20260925 14:59:35.163738 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.163647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54314)
0925 14:59:35.163686 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.163709 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 14:59:35.163846 32512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:45989: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:35.170980 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.163975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.9.141.125:43617)
0925 14:59:35.164016 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.164018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.164027 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:35.164056 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.164057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.164068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.164140 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.164555 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.165054 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.165826 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.166044 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.166134 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.168318 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.168319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.168320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.168326 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.169797 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.170223 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.170224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.170225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.170287 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.170502 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.170502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.170503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.170618 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.170679 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.170835 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.170912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:59:35.171010 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.163892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34618)
0925 14:59:35.163942 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.163953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.163998 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.164163 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.164165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.164333 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.164472 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.164475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.165160 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.165161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.165710 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.165712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.166134 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.166135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.166219 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.166717 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.166722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.168219 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.169883 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.169884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.169885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.170109 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.170368 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.170369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.170369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.170436 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.170683 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.170684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.170733 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.170805 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.170915 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:59:35.171789 32437 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:35.172160 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.172054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54324)
0925 14:59:35.172105 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.172139 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:59:35.172541 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.172443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54336)
0925 14:59:35.172495 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.172522 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:59:35.172693 32437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:45989: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
I20260925 14:59:35.203881 32437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } attempt: 1
I20260925 14:59:35.204486 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.204289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54338)
0925 14:59:35.204361 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.204451 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:59:35.204823 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.204722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54352)
0925 14:59:35.204773 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.204803 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:59:35.204998 32437 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:45989: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
W20260925 14:59:35.209810 32519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:59:35.210039 32519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:35.210060 32519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:35.210139 32519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:35.211508 32519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:35.211555 32519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:35.211570 32519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:35.211584 32519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:35.211598 32519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:35.213435 32519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.141.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.141.126:33651,127.9.141.125:43617,127.9.141.124:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/krb5kdc/kudu_127.9.141.124.keytab
--principal=kudu/127.9.141.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.141.124:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.141.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:59:35.213732 32519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:35.214012 32519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:35.219619 32525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:35.219784 32524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:35.219908 32519 server_base.cc:1034] running on GCE node
W20260925 14:59:35.219664 32527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:35.220299 32519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:35.220662 32519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:35.221876 32519 hybrid_clock.cc:648] HybridClock initialized: now 1790348375221861 us; error 32 us; skew 500 ppm
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:35.224812 32519 init.cc:377] Logged in from keytab as kudu/127.9.141.124@KRBTEST.COM (short username kudu)
I20260925 14:59:35.225356 32519 webserver.cc:460] Webserver started at http://127.9.141.124:35233/ using document root <none> and password file <none>
I20260925 14:59:35.225600 32519 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:35.225674 32519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:35.225821 32519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:59:35.226902 32519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data/instance:
uuid: "69447bdd363849899a6e9872889ca50a"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-1sv7"
server_key: "5a60c7c5f5d541bbb6ef8d7eaf95d486"
server_key_iv: "ce1cd71403b1e17a36fd17dbc3a8ad92"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.227286 32519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal/instance:
uuid: "69447bdd363849899a6e9872889ca50a"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-1sv7"
server_key: "5a60c7c5f5d541bbb6ef8d7eaf95d486"
server_key_iv: "ce1cd71403b1e17a36fd17dbc3a8ad92"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.228766 32519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:35.229614 32534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:35.229818 32519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:35.229924 32519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data,/tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal
uuid: "69447bdd363849899a6e9872889ca50a"
format_stamp: "Formatted at 2026-09-25 14:59:35 on dist-test-slave-1sv7"
server_key: "5a60c7c5f5d541bbb6ef8d7eaf95d486"
server_key_iv: "ce1cd71403b1e17a36fd17dbc3a8ad92"
server_key_version: "encryptionkey@0"
I20260925 14:59:35.230026 32519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:35.232024 32512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } attempt: 2
I20260925 14:59:35.232499 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.232330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54364)
0925 14:59:35.232411 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.232448 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:59:35.232903 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.232802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54378)
0925 14:59:35.232852 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.232878 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:59:35.233021 32512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.141.124:45989: Network error: Client connection negotiation failed: client connection to 127.9.141.124:45989: connect: Connection refused (error 111)
I20260925 14:59:35.249752 32519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:35.252202 32519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:35.252424 32519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:35.256739 32519 rpc_server.cc:307] RPC server started. Bound to: 127.9.141.124:45989
I20260925 14:59:35.256774 32586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.141.124:45989 every 8 connection(s)
I20260925 14:59:35.257153 32519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/data/info.pb
I20260925 14:59:35.258255 32587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:35.259439 9781 external_mini_cluster.cc:1388] Started /tmp/dist-test-task5afOW1/build/release/bin/kudu as pid 32519
I20260925 14:59:35.259650 9781 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task5afOW1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348105363960-9781-0/minicluster-data/master-2/wal/instance
I20260925 14:59:35.259914 9781 external_mini_cluster.cc:1428] Setting key 704aedefdfff6b919cc5a75485bffeac
I20260925 14:59:35.260602 32587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:59:35.269922 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.261453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.9.141.126:33651)
0925 14:59:35.261519 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.261522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.261665 (+ 143us) server_negotiation.cc:407] Connection header received
0925 14:59:35.261767 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.261768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.261779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.261844 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.262416 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.262893 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.263789 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.263999 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.264079 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.266754 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.266756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.266756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.266765 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.268452 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.268937 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.268938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.268939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.269005 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.269257 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.269258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.269258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.269385 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.269463 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.269743 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.269827 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:59:35.270128 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.261380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.126:33651 (local address 127.0.0.1:46966)
0925 14:59:35.261553 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.261591 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.261685 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.261863 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.261866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.262134 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.262314 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.262318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.262978 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.262979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.263683 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.263685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.264111 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.264112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.264242 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.264881 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.264901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.266647 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.268535 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.268542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.268545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.268849 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.269077 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.269078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.269079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.269190 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.269455 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.269461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.269611 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.269721 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.269879 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260925 14:59:35.270339 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.260698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.9.141.124:45989)
0925 14:59:35.260896 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.260905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.260939 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:59:35.261609 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.261618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.261678 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.261801 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.262435 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.263221 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.263999 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.264345 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.264462 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.267118 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.267126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.267127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.267135 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.269009 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.269399 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.269402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.269402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.269460 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.269674 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.269675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.269675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.269812 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.270068 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.270127 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.270215 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":148,"thread_start_us":140,"threads_started":1}
I20260925 14:59:35.270706 32587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.126@KRBTEST.COM
I20260925 14:59:35.277992 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.270995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.125:43617 (local address 127.0.0.1:34622)
0925 14:59:35.271133 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.271145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.271184 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.271318 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.271319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.271458 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.271536 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.271538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.272298 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.272299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.272871 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.272873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.273325 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.273326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.273384 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.273836 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.273842 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.275270 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.276946 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.276947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.276947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.277117 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.277340 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.277340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.277341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.277397 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.277662 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.277663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.277706 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.277788 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.277888 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 14:59:35.277999 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.271142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.9.141.125:43617)
0925 14:59:35.271195 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.271197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.271205 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:35.271233 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.271234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.271244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.271313 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.271620 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.272212 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.272956 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.273254 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.273327 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.275357 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.275358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.275358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.275366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.276844 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.277206 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.277208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.277208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.277262 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.277464 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.277466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.277466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.277584 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.277656 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.277815 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.277907 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:59:35.278616 32587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:59:35.278844 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.271108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.9.141.126:33651)
0925 14:59:35.271145 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.271147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.271254 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:59:35.271283 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.271284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.271292 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.271358 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.271837 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.272348 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.273071 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.273273 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.273346 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.275986 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.275987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.275988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.275994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.277539 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.277911 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.277913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.277913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.277973 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.278201 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.278202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.278203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.278344 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.278521 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.278563 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.278741 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:59:35.280118 32437 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } attempt: 2
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.124@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.126@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.141.125@KRBTEST.COM
I20260925 14:59:35.288626 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.279014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54384)
0925 14:59:35.279513 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.279524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.279570 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.280006 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.280007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.280166 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.280207 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.280209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.281168 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.281169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.281804 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.281806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.282738 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.282797 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.283497 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.283502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.284987 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.287254 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.287255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.287256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.287559 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.287798 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.287798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.287799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.287855 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.288178 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.288179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.288229 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.288311 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.288502 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482}
I20260925 14:59:35.289036 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.279469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.9.141.124:45989)
0925 14:59:35.279553 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.279555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.279566 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:59:35.279663 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.279664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.279675 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.279731 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.280288 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.281085 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.281890 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.282259 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.285079 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.285080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.285081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.285088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.286733 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.287657 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.287658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.287659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.287709 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.287936 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.287938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.287938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.288094 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.288178 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.288354 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.288911 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":448,"server-negotiator.queue_time_us":59}
I20260925 14:59:35.290088 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.280356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54390)
0925 14:59:35.280432 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.280442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.280476 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.280741 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.280743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.280938 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.280979 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.280981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.281750 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.281751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.282328 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.282331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.282738 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.282799 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.283597 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.283603 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.286094 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.288693 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.288694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.288694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.288925 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.289236 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.289237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.289238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.289309 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.289743 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.289744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.289794 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.289882 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.289988 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 14:59:35.290117 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.280489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.9.141.124:45989)
0925 14:59:35.280633 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.280635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.280642 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:59:35.280659 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.280660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.280669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.280733 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.281063 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.281668 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282420 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.282657 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282720 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.286194 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.286196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.286196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.286203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.288607 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.288993 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.288994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.288994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.289045 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.289536 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.289537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.289537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.289665 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.289748 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.289907 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.289988 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260925 14:59:35.290171 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.279888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.9.141.125:43617)
0925 14:59:35.279939 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.279949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.279957 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:35.279980 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.279981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.279990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.280039 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.280698 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.281404 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282343 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.282600 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.282650 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.286595 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.286598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.286598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.286608 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.288442 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.289138 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.289140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.289140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.289197 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.289456 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.289458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.289458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.289591 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.289772 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.290007 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.290084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:59:35.290846 32587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: Bootstrap starting.
I20260925 14:59:35.291676 32587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:35.291759 32437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478: Bootstrap starting.
I20260925 14:59:35.291854 9781 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:59:35.292013 32587 log.cc:824] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:35.292495 32437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:35.292750 32437 log.cc:824] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:35.292902 32587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: No bootstrap required, opened a new log
I20260925 14:59:35.293574 32437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478: No bootstrap required, opened a new log
I20260925 14:59:35.292891 32411 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "90e40cb46cec4ec988f5689c0e28b478"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980
I20260925 14:59:35.294523 32587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.294672 32587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69447bdd363849899a6e9872889ca50a, State: Initialized, Role: FOLLOWER
I20260925 14:59:35.294782 32587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.294919 32437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.295039 32437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e40cb46cec4ec988f5689c0e28b478, State: Initialized, Role: FOLLOWER
I20260925 14:59:35.295059 32595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.295154 32437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.295135 32587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:35.295230 32595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.295477 32437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:35.295444 32596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.295526 32411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:59:35.295670 32596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.295729 32411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:35.296660 32411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.296926 32411 leader_election.cc:288] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [CANDIDATE]: Term 1 election: Requested vote from peers a5256b21189e48a5a01c0f2a4037db86 (127.9.141.125:43617), 69447bdd363849899a6e9872889ca50a (127.9.141.124:45989)
W20260925 14:59:35.297628 32612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:35.297676 32612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 14:59:35.297786 32618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:35.297835 32618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:35.298710 32561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90e40cb46cec4ec988f5689c0e28b478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "69447bdd363849899a6e9872889ca50a"
I20260925 14:59:35.298833 32561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:35.299615 32561 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90e40cb46cec4ec988f5689c0e28b478 in term 1.
I20260925 14:59:35.299907 32388 leader_election.cc:302] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69447bdd363849899a6e9872889ca50a, 90e40cb46cec4ec988f5689c0e28b478; no voters:
I20260925 14:59:35.300033 32596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:35.300164 32596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [term 1 LEADER]: Becoming Leader. State: Replica: 90e40cb46cec4ec988f5689c0e28b478, State: Running, Role: LEADER
I20260925 14:59:35.300247 32596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.300661 32627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e40cb46cec4ec988f5689c0e28b478. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.300798 32627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:35.299242 32519 master_runner.cc:186] Error getting master registration for 127.9.141.125:43617: OK, instance_id { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" instance_seqno: 1790348375087661 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:59:35.301621 32519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:59:35.301409 32369 master_runner.cc:186] Error getting master registration for 127.9.141.125:43617: OK, instance_id { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" instance_seqno: 1790348375087661 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:59:35.303745 32369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:59:35.306366 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.298414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.9.141.125:43617)
0925 14:59:35.298509 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.298511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.298629 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:59:35.298941 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.298942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.298955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.298987 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.299816 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.300618 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.301648 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.301918 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.301955 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.303346 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.303348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.303348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.303355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.305052 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.305478 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.305479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.305480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.305553 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.305774 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.305775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.305775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.305932 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.306039 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.306193 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.306273 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:59:35.312858 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.298546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.9.141.124:45989)
0925 14:59:35.298701 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.298704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.304807 (+ 6103us) server_negotiation.cc:407] Connection header received
0925 14:59:35.304867 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.304869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.304884 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.304915 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.305591 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.306318 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.307473 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.307840 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.307916 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.309532 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.309534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.309534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.309543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.311415 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.311856 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.311857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.311858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.311931 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.312133 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.312134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.312134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.312270 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.312426 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.312642 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.312729 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 14:59:35.314832 32440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.304613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.9.141.126:33651)
0925 14:59:35.304657 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.304659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.306395 (+ 1736us) server_negotiation.cc:407] Connection header received
0925 14:59:35.306420 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.306422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.306434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.306483 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.308133 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.308859 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.309826 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.310134 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.310231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.311794 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.311796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.311796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.311803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.313462 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.313848 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.313850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.313850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.313907 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.314116 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.314117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.314117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.314232 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.314362 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.314457 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.314709 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:59:35.316598 32629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:35.316841 32629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:35.319146 32561 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [term 1 FOLLOWER]: Refusing update from remote peer 90e40cb46cec4ec988f5689c0e28b478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:35.319591 32596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:35.322237 32595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e40cb46cec4ec988f5689c0e28b478. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.322359 32595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.324146 32512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } attempt: 3
Sep 25 14:59:35 dist-test-slave-1sv7 krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.141.125@KRBTEST.COM for kudu/127.9.141.124@KRBTEST.COM
I20260925 14:59:35.332615 32596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.332727 32596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:35.332926 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.324503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.141.124:45989 (local address 127.0.0.1:54402)
0925 14:59:35.324545 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:35.324558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:35.324604 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:35.324779 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:35.324780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:35.324974 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:35.325027 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.325029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.325962 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.325963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.326785 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:35.326788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.327413 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.327414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:35.327470 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.327966 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:35.327972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:35.329491 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:35.331721 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.331722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.331723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.331956 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.332164 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:35.332164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:35.332165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:35.332220 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:35.332556 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:35.332557 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:35.332643 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 14:59:35.332708 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:59:35.332818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 14:59:35.333222 32592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:35.324530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.9.141.124:45989)
0925 14:59:35.324571 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.324573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.324588 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:59:35.324668 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.324669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.324680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.324732 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.325106 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.325868 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.326888 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:35.327323 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:35.327402 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:35.329592 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:35.329594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:35.329594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:35.329602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:35.331601 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.332039 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.332040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.332040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.332097 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:35.332282 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:35.332282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:35.332283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:35.332463 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:35.332527 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:35.333042 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 14:59:35.333123 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:59:35.333496 32629 catalog_manager.cc:1347] Generated new cluster ID: ccdcf46c700648da887d89d624333e71
I20260925 14:59:35.333550 32629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:35.333778 32595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.333851 32595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.334064 32631 mvcc.cc:205] Tried to move back new op lower bound from 7333266945304588288 to 7333266945230475264. Current Snapshot: MvccSnapshot[applied={T|T < 7333266945304588288 or (T in {7333266945304588288})}]
I20260925 14:59:35.334316 32632 mvcc.cc:205] Tried to move back new op lower bound from 7333266945304588288 to 7333266945230475264. Current Snapshot: MvccSnapshot[applied={T|T < 7333266945304588288 or (T in {7333266945304588288})}]
I20260925 14:59:35.337684 32512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: Bootstrap starting.
I20260925 14:59:35.338438 32512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:35.338729 32512 log.cc:824] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:35.340018 32512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: No bootstrap required, opened a new log
I20260925 14:59:35.342044 32512 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.342177 32512 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5256b21189e48a5a01c0f2a4037db86, State: Initialized, Role: FOLLOWER
I20260925 14:59:35.342252 32512 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } }
I20260925 14:59:35.342603 32512 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:35.342733 32486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90e40cb46cec4ec988f5689c0e28b478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5256b21189e48a5a01c0f2a4037db86"
I20260925 14:59:35.342875 32486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:35.343801 32486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90e40cb46cec4ec988f5689c0e28b478 in term 1.
I20260925 14:59:35.343962 32485 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [term 1 FOLLOWER]: Refusing update from remote peer 90e40cb46cec4ec988f5689c0e28b478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:35.344226 32627 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:35.352536 32651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.352669 32651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.352720 32651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e40cb46cec4ec988f5689c0e28b478. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.352782 32651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.352819 32651 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.352874 32651 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:59:35.353919 32649 mvcc.cc:205] Tried to move back new op lower bound from 7333266945304588288 to 7333266945230475264. Current Snapshot: MvccSnapshot[applied={T|T < 7333266945304588288}]
W20260925 14:59:35.354498 32652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:59:35.354559 32652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:59:35.358990 32627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90e40cb46cec4ec988f5689c0e28b478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e40cb46cec4ec988f5689c0e28b478" member_type: VOTER last_known_addr { host: "127.9.141.126" port: 33651 } } peers { permanent_uuid: "a5256b21189e48a5a01c0f2a4037db86" member_type: VOTER last_known_addr { host: "127.9.141.125" port: 43617 } } peers { permanent_uuid: "69447bdd363849899a6e9872889ca50a" member_type: VOTER last_known_addr { host: "127.9.141.124" port: 45989 } } }
I20260925 14:59:35.359103 32627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:35.361410 32629 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:35.362175 32629 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:35.365816 32629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e40cb46cec4ec988f5689c0e28b478: Generated new TSK 0
I20260925 14:59:35.366034 32629 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:59:35.405460 32440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.403309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.9.141.126:33651)
0925 14:59:35.403347 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.403349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.404885 (+ 1536us) server_negotiation.cc:407] Connection header received
0925 14:59:35.404984 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.404985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.404997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.405026 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.405382 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46998: BlockingRecv error: recv got EOF from 127.0.0.1:46998 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:59:35.405491 32592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.402670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.9.141.124:45989)
0925 14:59:35.402722 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.402725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.404589 (+ 1864us) server_negotiation.cc:407] Connection header received
0925 14:59:35.404675 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.404677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.404691 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.404750 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.405421 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54408: BlockingRecv error: recv got EOF from 127.0.0.1:54408 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:59:35.406090 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.403061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.9.141.125:43617)
0925 14:59:35.403108 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.403110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.404800 (+ 1690us) server_negotiation.cc:407] Connection header received
0925 14:59:35.404832 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.404833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.404845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.404895 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.406024 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34646: BlockingRecv error: recv got EOF from 127.0.0.1:34646 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 14:59:35.417657 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.416710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.9.141.125:43617)
0925 14:59:35.416762 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.416765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.417018 (+ 253us) server_negotiation.cc:407] Connection header received
0925 14:59:35.417124 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.417125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.417152 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.417210 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.417587 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34652: BlockingRecv error: recv got EOF from 127.0.0.1:34652 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 14:59:35.429672 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.428780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.9.141.125:43617)
0925 14:59:35.428828 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.428831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.429095 (+ 264us) server_negotiation.cc:407] Connection header received
0925 14:59:35.429185 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.429186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.429199 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.429262 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.429597 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34656: BlockingRecv error: recv got EOF from 127.0.0.1:34656 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 14:59:35.443142 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.433332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34664 (local address 127.9.141.125:43617)
0925 14:59:35.433375 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.433377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.442269 (+ 8892us) server_negotiation.cc:407] Connection header received
0925 14:59:35.442415 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.442417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.442432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.442467 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.443044 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34664: BlockingRecv error: recv got EOF from 127.0.0.1:34664 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:59:35.457217 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.452689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.9.141.125:43617)
0925 14:59:35.452738 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.452740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.456601 (+ 3861us) server_negotiation.cc:407] Connection header received
0925 14:59:35.456717 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.456718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.456731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.456762 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.457155 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34666: BlockingRecv error: recv got EOF from 127.0.0.1:34666 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 14:59:35.467486 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.466571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.9.141.125:43617)
0925 14:59:35.466617 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.466619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.466906 (+ 287us) server_negotiation.cc:407] Connection header received
0925 14:59:35.467013 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.467015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.467027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.467056 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.467424 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34678: BlockingRecv error: recv got EOF from 127.0.0.1:34678 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:59:35.486020 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.484947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34684 (local address 127.9.141.125:43617)
0925 14:59:35.484998 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.485000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.485352 (+ 352us) server_negotiation.cc:407] Connection header received
0925 14:59:35.485469 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.485470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.485483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.485515 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.485956 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34684: BlockingRecv error: recv got EOF from 127.0.0.1:34684 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 14:59:35.520669 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.519637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.9.141.125:43617)
0925 14:59:35.519690 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.519692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.519838 (+ 146us) server_negotiation.cc:407] Connection header received
0925 14:59:35.519901 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.519902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.519913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.519969 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.520595 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34690: BlockingRecv error: recv got EOF from 127.0.0.1:34690 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:59:35.586912 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.586085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.9.141.125:43617)
0925 14:59:35.586159 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.586161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.586173 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:59:35.586249 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.586250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.586261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.586336 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.586852 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34700: BlockingRecv error: recv got EOF from 127.0.0.1:34700 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 14:59:35.716796 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.716057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.9.141.125:43617)
0925 14:59:35.716129 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.716131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.716142 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:59:35.716175 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.716176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.716185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.716252 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.716698 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34702: BlockingRecv error: recv got EOF from 127.0.0.1:34702 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 14:59:35.975073 32516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:59:35.974127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.9.141.125:43617)
0925 14:59:35.974198 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:35.974199 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:35.974359 (+ 160us) server_negotiation.cc:407] Connection header received
0925 14:59:35.974442 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:35.974443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:35.974455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:35.974526 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:35.974974 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34716: BlockingRecv error: recv got EOF from 127.0.0.1:34716 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:59:36.298161 32612 catalog_manager.cc:1259] Loaded cluster ID: ccdcf46c700648da887d89d624333e71
I20260925 14:59:36.298242 32612 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: loading cluster ID for follower catalog manager: success
I20260925 14:59:36.299120 32612 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: acquiring CA information for follower catalog manager: success
I20260925 14:59:36.299454 32612 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69447bdd363849899a6e9872889ca50a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:59:36.355026 32652 catalog_manager.cc:1259] Loaded cluster ID: ccdcf46c700648da887d89d624333e71
I20260925 14:59:36.355151 32652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: loading cluster ID for follower catalog manager: success
I20260925 14:59:36.356000 32652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: acquiring CA information for follower catalog manager: success
I20260925 14:59:36.356340 32652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5256b21189e48a5a01c0f2a4037db86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:59:36.491389 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.488624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.9.141.125:43617)
0925 14:59:36.488748 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.488750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.488838 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:59:36.488920 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.488921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.488998 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.489091 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:36.489241 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.489753 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.490554 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.490759 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.490833 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.491013 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:36.491131 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:36.491225 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.491262 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.491284 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
I20260925 14:59:36.495914 32726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.493320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.9.141.124:45989)
0925 14:59:36.493461 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.493463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.493471 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:59:36.493508 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.493508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.493543 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.493625 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:36.493692 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.494170 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.494888 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.495087 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.495228 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.495528 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:36.495652 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:36.495714 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.495755 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.495778 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260925 14:59:36.500524 32726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.497652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.9.141.124:45989)
0925 14:59:36.497696 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.497699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.497877 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:59:36.497935 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.497936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.497948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.498000 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:36.498159 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.498724 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.499798 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.500035 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.500087 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.500249 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:36.500277 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:36.500315 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.500368 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.500426 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:59:36.501077 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.497620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.9.141.125:43617)
0925 14:59:36.497659 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.497661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.497762 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:59:36.497864 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.497865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.497878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.497908 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:36.498098 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.498824 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.499834 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.500054 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.500138 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.500788 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:36.500822 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:36.500849 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.500962 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.500987 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:36.501266 32732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:36.497615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.9.141.126:33651)
0925 14:59:36.497856 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:36.497858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:36.497974 (+ 116us) server_negotiation.cc:407] Connection header received
0925 14:59:36.498054 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:36.498055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:36.498106 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:36.498181 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:36.498355 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.499073 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.500174 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:36.500515 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:36.500572 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:36.500860 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:36.500989 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:36.501050 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:36.501154 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 14:59:36.501187 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":51,"threads_started":1}
I20260925 14:59:36.502911 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 32369
I20260925 14:59:36.508622 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 32444
I20260925 14:59:36.513507 9781 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task5afOW1/build/release/bin/kudu with pid 32519
2026-09-25T14:59:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2902 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5756 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271157 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST